{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "自然语言情感分析和文本匹配是日常生活中最常用的两类自然语言处理任务，本节主要介绍情感分析和文本匹配原理实现和典型模型，以及如何使用飞桨完成情感分析任务。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自然语言情感分析\n",
    "\n",
    "在学习本节课内容前，让我们先看一段来自肥伦秀的[视频片段](https://www.bilibili.com/video/av40396494?from=search&seid=9852893210841347755)，感受下人类语言情感的复杂性。众所周知，人类自然语言中包含了丰富的情感色彩：表达人的情绪（如悲伤、快乐）、表达人的心情（如倦怠、忧郁）、表达人的喜好（如喜欢、讨厌）、表达人的个性特征和表达人的立场等等。利用机器自动分析这些情感倾向，不但有助于帮助企业了解消费者对其产品的感受，为产品改进提供依据；同时还有助于企业分析商业伙伴们的态度，以便更好地进行商业决策。\n",
    "\n",
    "简单的说，我们可以将情感分析（sentiment classification）任务定义为一个分类问题，即指定一个文本输入，机器通过对文本进行分析、处理、归纳和推理后自动输出结论，如**图1**所示。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/1eaab515111a4dd6b4c9b71af37200c9df775a7958cf49ea88f638d6cef72775\" width=\"800\" ></center>\n",
    "<br><center>图1：情感分析任务</center></br>\n",
    "\n",
    "通常情况下，人们把情感分析任务看成一个三分类问题，如 **图2** 所示：\n",
    "<br></br>\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/988a21198ff84f3594562d212096f6357a7d587d7d1042b694d80a3e36eb876f\" width=\"600\" ></center>\n",
    "<br><center>图2：情感分析任务</center></br>\n",
    "\n",
    "- **正向：** 表示正面积极的情感，如高兴，幸福，惊喜，期待等。\n",
    "- **负向：** 表示负面消极的情感，如难过，伤心，愤怒，惊恐等。\n",
    "- **其他：** 其他类型的情感。\n",
    "\n",
    "在情感分析任务中，研究人员除了分析句子的情感类型外，还细化到以句子中具体的“方面”为分析主体进行情感分析（aspect-level），如下：\n",
    "\n",
    "> 这个薯片口味有点咸，太辣了，不过口感很脆。\n",
    "\n",
    "关于薯片的口味方面是一个负向评价（咸，太辣），然而对于口感方面却是一个正向评价（很脆）。\n",
    "\n",
    "> 我很喜欢夏威夷，就是这边的海鲜太贵了。\n",
    "\n",
    "关于夏威夷是一个正向评价（喜欢），然而对于夏威夷的海鲜却是一个负向评价（价格太贵）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用深度神经网络完成情感分析任务\n",
    "\n",
    "上一节课我们学习了通过把每个单词转换成向量的方式，可以完成单词语义计算任务。那么我们自然会联想到，是否可以把每个自然语言句子也转换成一个向量表示，并使用这个向量表示完成情感分析任务呢？\n",
    "\n",
    "在日常工作中有一个非常简单粗暴的解决方式：就是先把一个句子中所有词的embedding进行加和平均，再用得到的平均embedding作为整个句子的向量表示。然而由于自然语言变幻莫测，我们在使用神经网络处理句子的时候，往往会遇到如下两类问题：\n",
    "\n",
    "- **变长的句子：** 自然语言句子往往是变长的，不同的句子长度可能差别很大。然而大部分神经网络接受的输入都是张量，长度是固定的，那么如何让神经网络处理变长数据成为了一大挑战。\n",
    "\n",
    "- **组合的语义：** 自然语言句子往往对结构非常敏感，有时稍微颠倒单词的顺序都可能改变这句话的意思，比如：\n",
    "\n",
    "  >你等一下我做完作业就走。\n",
    "  >\n",
    "  >我等一下你做完工作就走。\n",
    "\n",
    "  >我不爱吃你做的饭。\n",
    "  >\n",
    "  >你不爱吃我做的饭。\n",
    "  \n",
    "  >我瞅你咋地。\n",
    "  >\n",
    "  >你瞅我咋滴。\n",
    "  \n",
    "因此，我们需要找到一个可以考虑词和词之间顺序（关系）的神经网络，用于更好地实现自然语言句子建模。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 处理变长数据\n",
    "\n",
    "在使用神经网络处理变长数据时，需要先设置一个全局变量max_seq_len，再对预料中的句子进行处理，将不同的句子组成mini-batch，用于神经网络学习和处理。\n",
    "\n",
    "**1. 设置全局变量**\n",
    "\n",
    "设定一个全局变量max_seq_len，用来控制神经网络最大可以处理文本的长度。我们可以先观察语料中句子的分布，再设置合理的max_seq_len值，以最高的性价比完成句子分类任务（如情感分类）。\n",
    "\n",
    "**2. 对语料中的句子进行处理**\n",
    "\n",
    "我们通常采用**截断+填充** 的方式，对语料中的句子进行处理，将不同的句子组成mini-batch，以便让句子转换成一个张量给神经网络进行处理计算，如 **图 3** 所示。\n",
    "<br></br>\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/1152814c654540f5a38028fbe50e4f2b279710360ebe4bb7830309148f22cb90\" width=\"400\" ></center>\n",
    "<br><center>图3：变长数据处理</center></br>\n",
    "\n",
    "* 对于长度超过max_seq_len的句子，我们通常会把这个句子进行截断，以便可以输入到一个张量中。句子截断的过程是有技巧的，有时截取句子的前一部分会比后一部分好，有时则恰好相反。当然也存在其他的截断方式，有兴趣的读者可以翻阅一下相关资料，这里不做赘述。\n",
    "* 对于句子长度不足max_seq_len的句子，我们一般会使用一个特殊的词语对这个句子进行填充，这个过程称为Padding。假设给定一个句子“我，爱，人工，智能”，max_seq_len=6，那么可能得到两种填充方式：\n",
    "  - **前向填充：** “[pad]，[pad]，我，爱，人工，智能”\n",
    "  - **后向填充：**“我，爱，人工，智能，[pad]，[pad]”\n",
    "\n",
    "同样，不同的填充方式也对网络训练效果有一定影响。一般来说，我们比较倾向选择后向填充的方式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 学习句子的语义\n",
    "\n",
    "上一节课我们已经学习了如何学习每个单词的语义信息，从刚才的举例中我们也会观察到，一个句子中词的顺序往往对这个句子的整体语义有比较重要的影响。因此，在刻画整个句子的语义信息过程中，不能撇开顺序信息。如果简单粗暴地把这个句子中所有词的向量做加和，会使得我们的模型无法区分句子的真实含义，例如：\n",
    "\n",
    ">我不爱吃你做的饭。\n",
    ">\n",
    ">你不爱吃我做的饭。\n",
    "\n",
    "一个有趣的想法，把一个自然语言句子看成一个序列，把整个自然语言的生成过程看成是一个序列生成的过程。例如对于句子“我，爱，人工，智能”，这句话的生成概率$\\text{P}(\\text{我，爱，人工，智能})$可以被表示为：\n",
    "\n",
    "$\\text{P}(\\text{我，爱，人工，智能})=\\text{P}(我|\\text{<s>})*\\text{P}(爱|\\text{<s>，我})* \\text{P}(人工|\\text{<s>，我，爱})* \\text{P}(智能|\\text{<s>，我，爱，人工})* \\text{P}(\\text{</s>}|\\text{<s>，我，爱，人工，智能})$\n",
    "\n",
    "其中$\\text{<s>}$和$\\text{</s>}$是两个特殊的不可见符合，表示一个句子在逻辑上的开始和结束。\n",
    "\n",
    "上面的公式把一个句子的生成过程建模成一个序列的决策过程，这就是香农在1950年左右提出的使用马尔可夫过程建模自然语言的思想。使用序列的视角看待和建模自然语言有一个明显的好处，那就是在对每个词建模的过程中，都有一个机会去学习这个词和之前生成的词之间的关系，并利用这种关系更好地处理自然语言。如 **图4** 所示，生成句子“我，爱，人工”后，“智能”在下一步生成的概率就变得很高了，因为“人工智能”经常同时出现。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/4ce2f35caaf44f90a80edc45f108b3f1365afab066234d8b863ce23daf93d12d\" width=\"700\" ></center>\n",
    "<br><center>图4：自然语言生成过程示意图</center></br>\n",
    "通过考虑句子内部的序列关系，我们就可以清晰地区分“我不爱吃你做的菜”和“你不爱吃我做的菜”这两句话之间的联系与不同了。事实上，目前大多数成功的自然语言模型都建立在对句子的序列化建模上。下面让我们学习一个经典的序列化建模模型：长短时记忆网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 循环神经网络RNN和长短时记忆网络LSTM\n",
    "\n",
    "循环神经网络（Recurrent Neural Network，RNN）是一个非常经典的面向序列的模型，可以对自然语言句子或是其他时序信号进行建模，RNN网络结构如 **图5** 所示。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/7b2e1ceabc7d44f484bfe81ddf404ac6013ca74d2b8443509b0cb1f06a87cd47\" width=\"600\" ></center>\n",
    "<br><center>图5：循环神经网络结构</center></br>\n",
    "\n",
    "不同于其他常见的神经网络结构，循环神经网络的输入是一个序列信息。假设给定任意一句话$[x_0, x_1, ..., x_n]$，其中每个$x_i$都代表了一个词，如“我，爱，人工，智能”。循环神经网络从左到右逐词阅读这个句子，并不断调用一个相同的RNN Cell来处理时序信息。每阅读一个单词，循环神经网络会先将本次输入的单词通过embedding lookup转换为一个向量表示。再把这个单词的向量表示和这个模型内部记忆的向量$h$融合起来，形成一个更新的记忆。最后将这个融合后的表示输出出来，作为它当前阅读到的所有内容的语义表示。当循环神经网络阅读过整个句子之后，我们就可以认为它的最后一个输出状态表示了整个句子的语义表示。\n",
    "\n",
    "听上去很复杂，下面我们以一个简单地例子来说明，假设输入的句子为：\n",
    "\n",
    "> “我，爱，人工，智能”\n",
    "\n",
    "循环神经网络开始从左到右阅读这个句子，在未经过任何阅读之前，循环神经网络中的记忆向量是空白的。其处理逻辑如下：\n",
    "1. 网络阅读单词“我”，并把单词“我”的向量表示和空白记忆相融合，输出一个向量$h_1$，用于表示“空白+我”的语义。\n",
    "1. 网络开始阅读单词“爱”，这时循环神经网络内部存在“空白+我”的记忆。循环神经网络会将“空白+我”和“爱”的向量表示相融合，并输出“空白+我+爱”的向量表示$h_2$，用于表示“我爱”这个短语的语义信息。\n",
    "1. 网络开始阅读单词“人工”，同样经过融合之后，输出“空白+我+爱+人工”的向量表示$h_3$，用于表示“空白+我+爱+人工”语义信息。\n",
    "1. 最终在网络阅读了“智能”单词后，便可以输出“我爱人工智能”这一句子的整体语义信息。\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "在实现当前输入$x_t$和已有记忆$h_{t-1}$融合的时候，循环神经网络采用相加并通过一个激活函数tanh的方式实现：$h_t = tanh(WX_{t}+VH_{t-1}+b)$\n",
    "\n",
    "tanh函数是一个值域为（-1,1）的函数，其作用是长期维持内部记忆在一个固定的数值范围内，防止因多次迭代更新导致数值爆炸。同时tanh的导数是一个平滑的函数，会让神经网络的训练变得更加简单。\n",
    "\n",
    "-----\n",
    "\n",
    "上述方法听上去很有效（事实上在有些任务上效果还不错），但是存在一个明显的缺陷，就是当阅读很长的序列时，网络内部的信息会变得越来越复杂，甚至会超过网络的记忆能力，使得最终的输出信息变得混乱无用。长短时记忆网络（Long Short-Term Memory，LSTM）内部的复杂结构正是为处理这类问题而设计的，其网络结构如 **图6** 所示。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/edca17c371154b0ea375a8902c9d055011848057c9524049afaa4f65f99158af\" width=\"600\" ></center>\n",
    "<br><center>图6：LSTM网络结构</center></br>\n",
    "\n",
    "长短时记忆网络的结构和循环神经网络非常类似，都是通过不断调用同一个cell来逐次处理时序信息。每阅读一个新单词$x_t$，就会输出一个新的输出信号$h_t$，用来表示当前阅读到所有内容的整体向量表示。不过二者又有一个明显区别，长短时记忆网络在不同cell之间传递的是二个记忆信息，而不像循环神经网络一样只有一个记忆信息，此外长短时记忆网络的内部结构也更加复杂，如 **图7** 所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/03363bd09f5143ba86f879e18a81ea889b25ffe6f55c4151b92c06619c782b0f\" width=\"600\" ></center>\n",
    "<br><center>图7：LSTM网络内部结构示意图</center></br>\n",
    "\n",
    "区别于循环神经网络RNN，长短时记忆网络最大的特点是在更新内部记忆时，引入了遗忘机制。即容许网络忘记过去阅读过程中看到的一些无关紧要的信息，只保留有用的历史信息。通过这种方式延长了记忆长度。举个例子：\n",
    "\n",
    "> 我觉得这家餐馆的菜品很不错，烤鸭非常正宗，包子也不错，酱牛肉很有嚼劲。但是服务员态度太恶劣了，我们在门口等了50分钟都没有能成功进去，好不容易进去了，桌子也半天没人打扫。整个环境非常吵闹，我的孩子都被吓哭了，我下次不会带朋友来。\n",
    "\n",
    "当我们阅读上面这段话的时候，可能会记住一些关键词，如烤鸭好吃、牛肉有嚼劲、环境吵等，但也会忽略一些不重要的内容，如“我觉得”、“好不容易”等，长短时记忆网络正是受这个启发而设计的。\n",
    "\n",
    "长短时记忆网络的Cell有三个输入：\n",
    "\n",
    "- 这个网络新看到的输入信号，如下一个单词，记为$x_{t}$， 其中$x_{t}$是一个向量，$t$代表了当前时刻。\n",
    "- 这个网络在上一步的输出信号，记为$h_{t-1}$，这是一个向量，维度同$x_{t}$相同。\n",
    "- 这个网络在上一步的记忆信号，记为$c_{t-1}$，这是一个向量，维度同$x_{t}$相同。\n",
    "\n",
    "得到这两个信号之后，长短时记忆网络没有立即去融合这两个向量，而是计算了如下门的权重：\n",
    "\n",
    "- 输入门：$i_{t}=sigmoid(W_{i}X_{t}+V_{i}H_{t-1}+b_i)$，控制有多少输入信号会被融合。\n",
    "- 遗忘门：$f_{t}=sigmoid(W_{f}X_{t}+V_{f}H_{t-1}+b_f)$，控制有多少过去的记忆会被融合。\n",
    "- 输出门：$o_{t}=sigmoid(W_{o}X_{t}+V_{o}H_{t-1}+b_o)$，控制最终输出多少记忆。\n",
    "- 单元状态：$g_{t}=tanh(W_{g}X_{t}+V_{g}H_{t-1}+b_g)$\n",
    "\n",
    "通过学习这些门的权重设置，长短时记忆网络可以根据当前的输入信号和记忆信息，有选择性地忽略或者强化当前的记忆或是输入信号，帮助网络更好地学习长句子的语义信息：\n",
    "\n",
    "$c_{t} = f_{t} \\cdot c_{t-1} + i_{t} \\cdot g_{t}$\n",
    "\n",
    "$h_{t} = o_{t} \\cdot tanh(c_{t})$\n",
    "\n",
    "------\n",
    "\n",
    "**说明：**\n",
    "\n",
    "事实上，长短时记忆网络之所以能更好地对长文本进行建模，还存在另外一套更加严谨的计算和证明，有兴趣的读者可以翻阅一下引文中的参考资料进行详细研究。\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用LSTM完成情感分析任务\n",
    "\n",
    "借助长短时记忆网络，我们可以非常轻松地完成情感分析任务。如 **图8** 所示。对于每个句子，我们首先通过截断和填充的方式，把这些句子变成固定长度的向量。然后，利用长短时记忆网络，从左到右开始阅读每个句子。在完成阅读之后，我们使用长短时记忆网络的最后一个输出记忆，作为整个句子的语义信息，并直接把这个向量作为输入，送入一个分类层进行分类，从而完成对情感分析问题的神经网络建模。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/c16a69f210b44c7c8f4b977a695f3c8f3b551a7798114cdf98118cf26cffdc16\" width=\"400\" ></center>\n",
    "<br><center>图8：LSTM完成情感分析任务流程</center></br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用飞桨实现基于LSTM的情感分析模型 \n",
    "\n",
    "接下来让我们看看如何使用飞桨实现一个基于长短时记忆网络的情感分析模型。在飞桨中，不同深度学习模型的训练过程基本一致，流程如下：\n",
    "\n",
    "1. **数据处理**：选择需要使用的数据，并做好必要的预处理工作。\n",
    "\n",
    "2. **网络定义**：使用飞桨定义好网络结构，包括输入层，中间层，输出层，损失函数和优化算法。\n",
    "\n",
    "3. **网络训练**：将准备好的数据送入神经网络进行学习，并观察学习的过程是否正常，如损失函数值是否在降低，也可以打印一些中间步骤的结果出来等。\n",
    "\n",
    "4. **网络评估**：使用测试集合测试训练好的神经网络，看看训练效果如何。\n",
    "\n",
    "在数据处理前，需要先加载飞桨平台（如果用户在本地使用，请确保已经安装飞桨）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import os\n",
    "import re\n",
    "import sys\n",
    "import six\n",
    "import requests\n",
    "import string\n",
    "import tarfile\n",
    "import hashlib\n",
    "from collections import OrderedDict \n",
    "import math\n",
    "import random\n",
    "import numpy as np\n",
    "import paddle\n",
    "import paddle.fluid as fluid\n",
    "\n",
    "from paddle.fluid.dygraph.nn import Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据处理\n",
    "\n",
    "首先，需要下载语料用于模型训练和评估效果。我们使用的是IMDB的电影评论数据，这个数据集是一个开源的英文数据集，由训练数据和测试数据组成。每个数据都分别由若干小文件组成，每个小文件内部都是一段用户关于某个电影的真实评价，以及他/她对这个电影的情感倾向（是正向还是负向），数据集下载的代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download():\n",
    "    #通过python的requests类，下载存储在\n",
    "    #https://dataset.bj.bcebos.com/imdb%2FaclImdb_v1.tar.gz的文件\n",
    "    corpus_url = \"https://dataset.bj.bcebos.com/imdb%2FaclImdb_v1.tar.gz\"\n",
    "    web_request = requests.get(corpus_url)\n",
    "    corpus = web_request.content\n",
    "\n",
    "    #将下载的文件写在当前目录的aclImdb_v1.tar.gz文件内\n",
    "    with open(\"./aclImdb_v1.tar.gz\", \"wb\") as f:\n",
    "        f.write(corpus)\n",
    "    f.close()\n",
    "\n",
    "download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，将数据集加载到程序中，并打印一小部分数据观察一下数据集的特点，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentence 0, Zentropa has much in common with The Third Man, another noir-like film set among the rubble of postwar Europe. Like TTM, there is much inventive camera work. There is an innocent American who gets emotionally involved with a woman he doesn't really understand, and whose naivety is all the more striking in contrast with the natives.<br /><br />But I'd have to say that The Third Man has a more well-crafted storyline. Zentropa is a bit disjointed in this respect. Perhaps this is intentional: it is presented as a dream/nightmare, and making it too coherent would spoil the effect. <br /><br />This movie is unrelentingly grim--\"noir\" in more than one sense; one never sees the sun shine. Grim, but intriguing, and frightening.\n",
      "sentence 0, label 1\n",
      "sentence 1, Zentropa is the most original movie I've seen in years. If you like unique thrillers that are influenced by film noir, then this is just the right cure for all of those Hollywood summer blockbusters clogging the theaters these days. Von Trier's follow-ups like Breaking the Waves have gotten more acclaim, but this is really his best work. It is flashy without being distracting and offers the perfect combination of suspense and dark humor. It's too bad he decided handheld cameras were the wave of the future. It's hard to say who talked him away from the style he exhibits here, but it's everyone's loss that he went into his heavily theoretical dogma direction instead.\n",
      "sentence 1, label 1\n",
      "sentence 2, Lars Von Trier is never backward in trying out new techniques. Some of them are very original while others are best forgotten.<br /><br />He depicts postwar Germany as a nightmarish train journey. With so many cities lying in ruins, Leo Kessler a young American of German descent feels obliged to help in their restoration. It is not a simple task as he quickly finds out.<br /><br />His uncle finds him a job as a night conductor on the Zentropa Railway Line. His job is to attend to the needs of the passengers. When the shoes are polished a chalk mark is made on the soles. A terrible argument ensues when a passenger's shoes are not chalked despite the fact they have been polished. There are many allusions to the German fanaticism of adherence to such stupid details.<br /><br />The railway journey is like an allegory representing man's procession through life with all its trials and tribulations. In one sequence Leo dashes through the back carriages to discover them filled with half-starved bodies appearing to have just escaped from Auschwitz . These images, horrible as they are, are fleeting as in a dream, each with its own terrible impact yet unconnected.<br /><br />At a station called Urmitz Leo jumps from the train with a parceled bomb. In view of many by-standers he connects the bomb to the underside of a carriage. He returns to his cabin and makes a connection to a time clock. Later he jumps from the train (at high speed) and lies in the cool grass on a river bank. Looking at the stars above he decides that his job is to build and not destroy. Subsequently as he sees the train approaching a giant bridge he runs at breakneck speed to board the train and stop the clock. If you care to analyse the situation it is a completely impossible task. Quite ridiculous in fact. It could only happen in a dream.<br /><br />It's strange how one remembers little details such as a row of cups hanging on hooks and rattling away with the swaying of the train.<br /><br />Despite the fact that this film is widely acclaimed, I prefer Lars Von Trier's later films (Breaking the Waves and The Idiots). The bomb scene described above really put me off. Perhaps I'm a realist.\n",
      "sentence 2, label 1\n",
      "sentence 3, *Contains spoilers due to me having to describe some film techniques, so read at your own risk!*<br /><br />I loved this film. The use of tinting in some of the scenes makes it seem like an old photograph come to life. I also enjoyed the projection of people on a back screen. For instance, in one scene, Leopold calls his wife and she is projected behind him rather than in a typical split screen. Her face is huge in the back and Leo's is in the foreground.<br /><br />One of the best uses of this is when the young boys kill the Ravensteins on the train, a scene shot in an almost political poster style, with facial close ups. It reminded me of Battleship Potemkin, that intense constant style coupled with the spray of red to convey tons of horror without much gore. Same with the scene when Katharina finds her father dead in the bathtub...you can only see the red water on the side. It is one of the things I love about Von Trier, his understatement of horror, which ends up making it all the more creepy.<br /><br />The use of text in the film was unique, like when Leo's character is pushed by the word, \"Werewolf.\" I have never seen anything like that in a film.<br /><br />The use of black comedy in this film was well done. Ernst-Hugo Järegård is great as Leo's uncle. It brings up the snickers I got from his role in the Kingdom (Riget.) This humor makes the plotline of absurd anal retentiveness of train conductors against the terrible backdrop of WW2 and all the chaos, easier to take. It reminds me of Riget in the way the hospital administrator is trying to maintain a normalcy at the end of part one when everything is going crazy. It shows that some people are truly oblivious to the awful things happening around them. Yet some people, like Leo, are tuned in, but do nothing positive about it.<br /><br />The voice over, done expertly well by Max von Sydow, is amusing too. It draws you into the story and makes you jump into Leo's head, which at times is a scary place to be.<br /><br />The movie brings up the point that one is a coward if they don't choose a side. I see the same idea used in Dancer in the Dark, where Bjork's character doesn't speak up for herself and ends up being her own destruction. Actually, at one time, Von Trier seemed anti-woman to me, by making Breaking the Waves and Dancer, but now I know his male characters don't fare well either! I found myself at the same place during the end of Dancer, when you seriously want the main character to rethink their actions, but of course, they never do!\n",
      "sentence 3, label 1\n",
      "sentence 4, That was the first thing that sprang to mind as I watched the closing credits to Europa make there was across the screen, never in my entire life have I seen a film of such technical genius, the visuals of Europa are so impressive that any film I watch in it's wake will only pale in comparison, forget your Michael Bay, Ridley Scott slick Hollywood cinematography, Europa has more ethereal beauty than anything those two could conjure up in a million years. Now I'd be the first to hail Lars von Trier a genius just off the back of his films Breaking the Waves and Dancer in the Dark, but this is stupid, the fact that Europa has gone un-noticed by film experts for so long is a crime against cinema, whilst overrated rubbish like Crouching Tiger, Hidden Dragon and Life is Beautiful clean up at the academy awards (but what do the know) Europa has been hidden away, absent form video stores and (until recently) any British TV channels. <br /><br />The visuals in Europa are not MTV gloss; it's not a case of style over substance, its more a case of substance dictating style. Much like his first film The Element of Crime, von Trier uses the perspective of the main character to draw us into his world, and much like Element, the film begins with the main character (or in the case of Europa, we the audience) being hypnotized. As we move down the tracks, the voice of the Narrator (Max von Sydow) counts us down into a deep sleep, until we awake in Europa. This allows von Trier and his three cinematographers to pay with the conventions of time and imagery, there are many scenes in Europa when a character in the background, who is in black and white, will interact with a person in the foreground who will be colour, von Trier is trying to show us how much precedence the coloured item or person has over the plot, for instance, it's no surprise that the first shot of Leopold Kessler (Jean-marc Barr) is in colour, since he is the only character who's actions have superiority over the film. <br /><br />The performances are good, they may not be on par with performances in later von Trier films, but that's just because the images are sometimes so distracting that you don't really pick up on them the first time round. But I would like to point out the fantastic performance of Jean-Marc Barr in the lead role, whose blind idealism is slowly warn down by the two opposing sides, until he erupts in the films final act. Again, muck like The Element of Crime, the film ends with our hero unable to wake up from his nightmare state, left in this terrible place, with only the continuing narration of von Sydow to seal his fate. Europa is a tremendous film, and I cant help thinking what a shame that von Trier has abandoned this way of filming, since he was clearly one of the most talented visual directors working at that time, Europa, much like the rest of his cinematic cannon is filled with a wealth of iconic scenes. His dedication to composition and mise-en-scene is unrivalled, not to mention his use of sound and production design. But since his no-frills melodramas turned out to be Breaking the Waves and Dancer in the Dark then who can argue, but it does seems like a waste of an imaginative talent. 10/10\n",
      "sentence 4, label 1\n"
     ]
    }
   ],
   "source": [
    "def load_imdb(is_training):\n",
    "    data_set = []\n",
    "\n",
    "    #aclImdb_v1.tar.gz解压后是一个目录\n",
    "    #我们可以使用python的rarfile库进行解压\n",
    "    #训练数据和测试数据已经经过切分，其中训练数据的地址为：\n",
    "    #./aclImdb/train/pos/ 和 ./aclImdb/train/neg/，分别存储着正向情感的数据和负向情感的数据\n",
    "    #我们把数据依次读取出来，并放到data_set里\n",
    "    #data_set中每个元素都是一个二元组，（句子，label），其中label=0表示负向情感，label=1表示正向情感\n",
    "    \n",
    "    for label in [\"pos\", \"neg\"]:\n",
    "        with tarfile.open(\"./aclImdb_v1.tar.gz\") as tarf:\n",
    "            path_pattern = \"aclImdb/train/\" + label + \"/.*\\.txt$\" if is_training \\\n",
    "                else \"aclImdb/test/\" + label + \"/.*\\.txt$\"\n",
    "            path_pattern = re.compile(path_pattern)\n",
    "            tf = tarf.next()\n",
    "            while tf != None:\n",
    "                if bool(path_pattern.match(tf.name)):\n",
    "                    sentence = tarf.extractfile(tf).read().decode()\n",
    "                    sentence_label = 0 if label == 'neg' else 1\n",
    "                    data_set.append((sentence, sentence_label)) \n",
    "                tf = tarf.next()\n",
    "\n",
    "    return data_set\n",
    "\n",
    "train_corpus = load_imdb(True)\n",
    "test_corpus = load_imdb(False)\n",
    "\n",
    "for i in range(5):\n",
    "    print(\"sentence %d, %s\" % (i, train_corpus[i][0]))    \n",
    "    print(\"sentence %d, label %d\" % (i, train_corpus[i][1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般来说，在自然语言处理中，需要先对语料进行切词，这里我们可以使用空格把每个句子切成若干词的序列，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['zentropa', 'has', 'much', 'in', 'common', 'with', 'the', 'third', 'man,', 'another', 'noir-like', 'film', 'set', 'among', 'the', 'rubble', 'of', 'postwar', 'europe.', 'like', 'ttm,', 'there', 'is', 'much', 'inventive', 'camera', 'work.', 'there', 'is', 'an', 'innocent', 'american', 'who', 'gets', 'emotionally', 'involved', 'with', 'a', 'woman', 'he', \"doesn't\", 'really', 'understand,', 'and', 'whose', 'naivety', 'is', 'all', 'the', 'more', 'striking', 'in', 'contrast', 'with', 'the', 'natives.<br', '/><br', '/>but', \"i'd\", 'have', 'to', 'say', 'that', 'the', 'third', 'man', 'has', 'a', 'more', 'well-crafted', 'storyline.', 'zentropa', 'is', 'a', 'bit', 'disjointed', 'in', 'this', 'respect.', 'perhaps', 'this', 'is', 'intentional:', 'it', 'is', 'presented', 'as', 'a', 'dream/nightmare,', 'and', 'making', 'it', 'too', 'coherent', 'would', 'spoil', 'the', 'effect.', '<br', '/><br', '/>this', 'movie', 'is', 'unrelentingly', 'grim--\"noir\"', 'in', 'more', 'than', 'one', 'sense;', 'one', 'never', 'sees', 'the', 'sun', 'shine.', 'grim,', 'but', 'intriguing,', 'and', 'frightening.'], 1), (['zentropa', 'is', 'the', 'most', 'original', 'movie', \"i've\", 'seen', 'in', 'years.', 'if', 'you', 'like', 'unique', 'thrillers', 'that', 'are', 'influenced', 'by', 'film', 'noir,', 'then', 'this', 'is', 'just', 'the', 'right', 'cure', 'for', 'all', 'of', 'those', 'hollywood', 'summer', 'blockbusters', 'clogging', 'the', 'theaters', 'these', 'days.', 'von', \"trier's\", 'follow-ups', 'like', 'breaking', 'the', 'waves', 'have', 'gotten', 'more', 'acclaim,', 'but', 'this', 'is', 'really', 'his', 'best', 'work.', 'it', 'is', 'flashy', 'without', 'being', 'distracting', 'and', 'offers', 'the', 'perfect', 'combination', 'of', 'suspense', 'and', 'dark', 'humor.', \"it's\", 'too', 'bad', 'he', 'decided', 'handheld', 'cameras', 'were', 'the', 'wave', 'of', 'the', 'future.', \"it's\", 'hard', 'to', 'say', 'who', 'talked', 'him', 'away', 'from', 'the', 'style', 'he', 'exhibits', 'here,', 'but', \"it's\", \"everyone's\", 'loss', 'that', 'he', 'went', 'into', 'his', 'heavily', 'theoretical', 'dogma', 'direction', 'instead.'], 1), (['lars', 'von', 'trier', 'is', 'never', 'backward', 'in', 'trying', 'out', 'new', 'techniques.', 'some', 'of', 'them', 'are', 'very', 'original', 'while', 'others', 'are', 'best', 'forgotten.<br', '/><br', '/>he', 'depicts', 'postwar', 'germany', 'as', 'a', 'nightmarish', 'train', 'journey.', 'with', 'so', 'many', 'cities', 'lying', 'in', 'ruins,', 'leo', 'kessler', 'a', 'young', 'american', 'of', 'german', 'descent', 'feels', 'obliged', 'to', 'help', 'in', 'their', 'restoration.', 'it', 'is', 'not', 'a', 'simple', 'task', 'as', 'he', 'quickly', 'finds', 'out.<br', '/><br', '/>his', 'uncle', 'finds', 'him', 'a', 'job', 'as', 'a', 'night', 'conductor', 'on', 'the', 'zentropa', 'railway', 'line.', 'his', 'job', 'is', 'to', 'attend', 'to', 'the', 'needs', 'of', 'the', 'passengers.', 'when', 'the', 'shoes', 'are', 'polished', 'a', 'chalk', 'mark', 'is', 'made', 'on', 'the', 'soles.', 'a', 'terrible', 'argument', 'ensues', 'when', 'a', \"passenger's\", 'shoes', 'are', 'not', 'chalked', 'despite', 'the', 'fact', 'they', 'have', 'been', 'polished.', 'there', 'are', 'many', 'allusions', 'to', 'the', 'german', 'fanaticism', 'of', 'adherence', 'to', 'such', 'stupid', 'details.<br', '/><br', '/>the', 'railway', 'journey', 'is', 'like', 'an', 'allegory', 'representing', \"man's\", 'procession', 'through', 'life', 'with', 'all', 'its', 'trials', 'and', 'tribulations.', 'in', 'one', 'sequence', 'leo', 'dashes', 'through', 'the', 'back', 'carriages', 'to', 'discover', 'them', 'filled', 'with', 'half-starved', 'bodies', 'appearing', 'to', 'have', 'just', 'escaped', 'from', 'auschwitz', '.', 'these', 'images,', 'horrible', 'as', 'they', 'are,', 'are', 'fleeting', 'as', 'in', 'a', 'dream,', 'each', 'with', 'its', 'own', 'terrible', 'impact', 'yet', 'unconnected.<br', '/><br', '/>at', 'a', 'station', 'called', 'urmitz', 'leo', 'jumps', 'from', 'the', 'train', 'with', 'a', 'parceled', 'bomb.', 'in', 'view', 'of', 'many', 'by-standers', 'he', 'connects', 'the', 'bomb', 'to', 'the', 'underside', 'of', 'a', 'carriage.', 'he', 'returns', 'to', 'his', 'cabin', 'and', 'makes', 'a', 'connection', 'to', 'a', 'time', 'clock.', 'later', 'he', 'jumps', 'from', 'the', 'train', '(at', 'high', 'speed)', 'and', 'lies', 'in', 'the', 'cool', 'grass', 'on', 'a', 'river', 'bank.', 'looking', 'at', 'the', 'stars', 'above', 'he', 'decides', 'that', 'his', 'job', 'is', 'to', 'build', 'and', 'not', 'destroy.', 'subsequently', 'as', 'he', 'sees', 'the', 'train', 'approaching', 'a', 'giant', 'bridge', 'he', 'runs', 'at', 'breakneck', 'speed', 'to', 'board', 'the', 'train', 'and', 'stop', 'the', 'clock.', 'if', 'you', 'care', 'to', 'analyse', 'the', 'situation', 'it', 'is', 'a', 'completely', 'impossible', 'task.', 'quite', 'ridiculous', 'in', 'fact.', 'it', 'could', 'only', 'happen', 'in', 'a', 'dream.<br', '/><br', \"/>it's\", 'strange', 'how', 'one', 'remembers', 'little', 'details', 'such', 'as', 'a', 'row', 'of', 'cups', 'hanging', 'on', 'hooks', 'and', 'rattling', 'away', 'with', 'the', 'swaying', 'of', 'the', 'train.<br', '/><br', '/>despite', 'the', 'fact', 'that', 'this', 'film', 'is', 'widely', 'acclaimed,', 'i', 'prefer', 'lars', 'von', \"trier's\", 'later', 'films', '(breaking', 'the', 'waves', 'and', 'the', 'idiots).', 'the', 'bomb', 'scene', 'described', 'above', 'really', 'put', 'me', 'off.', 'perhaps', \"i'm\", 'a', 'realist.'], 1), (['*contains', 'spoilers', 'due', 'to', 'me', 'having', 'to', 'describe', 'some', 'film', 'techniques,', 'so', 'read', 'at', 'your', 'own', 'risk!*<br', '/><br', '/>i', 'loved', 'this', 'film.', 'the', 'use', 'of', 'tinting', 'in', 'some', 'of', 'the', 'scenes', 'makes', 'it', 'seem', 'like', 'an', 'old', 'photograph', 'come', 'to', 'life.', 'i', 'also', 'enjoyed', 'the', 'projection', 'of', 'people', 'on', 'a', 'back', 'screen.', 'for', 'instance,', 'in', 'one', 'scene,', 'leopold', 'calls', 'his', 'wife', 'and', 'she', 'is', 'projected', 'behind', 'him', 'rather', 'than', 'in', 'a', 'typical', 'split', 'screen.', 'her', 'face', 'is', 'huge', 'in', 'the', 'back', 'and', \"leo's\", 'is', 'in', 'the', 'foreground.<br', '/><br', '/>one', 'of', 'the', 'best', 'uses', 'of', 'this', 'is', 'when', 'the', 'young', 'boys', 'kill', 'the', 'ravensteins', 'on', 'the', 'train,', 'a', 'scene', 'shot', 'in', 'an', 'almost', 'political', 'poster', 'style,', 'with', 'facial', 'close', 'ups.', 'it', 'reminded', 'me', 'of', 'battleship', 'potemkin,', 'that', 'intense', 'constant', 'style', 'coupled', 'with', 'the', 'spray', 'of', 'red', 'to', 'convey', 'tons', 'of', 'horror', 'without', 'much', 'gore.', 'same', 'with', 'the', 'scene', 'when', 'katharina', 'finds', 'her', 'father', 'dead', 'in', 'the', 'bathtub...you', 'can', 'only', 'see', 'the', 'red', 'water', 'on', 'the', 'side.', 'it', 'is', 'one', 'of', 'the', 'things', 'i', 'love', 'about', 'von', 'trier,', 'his', 'understatement', 'of', 'horror,', 'which', 'ends', 'up', 'making', 'it', 'all', 'the', 'more', 'creepy.<br', '/><br', '/>the', 'use', 'of', 'text', 'in', 'the', 'film', 'was', 'unique,', 'like', 'when', \"leo's\", 'character', 'is', 'pushed', 'by', 'the', 'word,', '\"werewolf.\"', 'i', 'have', 'never', 'seen', 'anything', 'like', 'that', 'in', 'a', 'film.<br', '/><br', '/>the', 'use', 'of', 'black', 'comedy', 'in', 'this', 'film', 'was', 'well', 'done.', 'ernst-hugo', 'järegård', 'is', 'great', 'as', \"leo's\", 'uncle.', 'it', 'brings', 'up', 'the', 'snickers', 'i', 'got', 'from', 'his', 'role', 'in', 'the', 'kingdom', '(riget.)', 'this', 'humor', 'makes', 'the', 'plotline', 'of', 'absurd', 'anal', 'retentiveness', 'of', 'train', 'conductors', 'against', 'the', 'terrible', 'backdrop', 'of', 'ww2', 'and', 'all', 'the', 'chaos,', 'easier', 'to', 'take.', 'it', 'reminds', 'me', 'of', 'riget', 'in', 'the', 'way', 'the', 'hospital', 'administrator', 'is', 'trying', 'to', 'maintain', 'a', 'normalcy', 'at', 'the', 'end', 'of', 'part', 'one', 'when', 'everything', 'is', 'going', 'crazy.', 'it', 'shows', 'that', 'some', 'people', 'are', 'truly', 'oblivious', 'to', 'the', 'awful', 'things', 'happening', 'around', 'them.', 'yet', 'some', 'people,', 'like', 'leo,', 'are', 'tuned', 'in,', 'but', 'do', 'nothing', 'positive', 'about', 'it.<br', '/><br', '/>the', 'voice', 'over,', 'done', 'expertly', 'well', 'by', 'max', 'von', 'sydow,', 'is', 'amusing', 'too.', 'it', 'draws', 'you', 'into', 'the', 'story', 'and', 'makes', 'you', 'jump', 'into', \"leo's\", 'head,', 'which', 'at', 'times', 'is', 'a', 'scary', 'place', 'to', 'be.<br', '/><br', '/>the', 'movie', 'brings', 'up', 'the', 'point', 'that', 'one', 'is', 'a', 'coward', 'if', 'they', \"don't\", 'choose', 'a', 'side.', 'i', 'see', 'the', 'same', 'idea', 'used', 'in', 'dancer', 'in', 'the', 'dark,', 'where', \"bjork's\", 'character', \"doesn't\", 'speak', 'up', 'for', 'herself', 'and', 'ends', 'up', 'being', 'her', 'own', 'destruction.', 'actually,', 'at', 'one', 'time,', 'von', 'trier', 'seemed', 'anti-woman', 'to', 'me,', 'by', 'making', 'breaking', 'the', 'waves', 'and', 'dancer,', 'but', 'now', 'i', 'know', 'his', 'male', 'characters', \"don't\", 'fare', 'well', 'either!', 'i', 'found', 'myself', 'at', 'the', 'same', 'place', 'during', 'the', 'end', 'of', 'dancer,', 'when', 'you', 'seriously', 'want', 'the', 'main', 'character', 'to', 'rethink', 'their', 'actions,', 'but', 'of', 'course,', 'they', 'never', 'do!'], 1), (['that', 'was', 'the', 'first', 'thing', 'that', 'sprang', 'to', 'mind', 'as', 'i', 'watched', 'the', 'closing', 'credits', 'to', 'europa', 'make', 'there', 'was', 'across', 'the', 'screen,', 'never', 'in', 'my', 'entire', 'life', 'have', 'i', 'seen', 'a', 'film', 'of', 'such', 'technical', 'genius,', 'the', 'visuals', 'of', 'europa', 'are', 'so', 'impressive', 'that', 'any', 'film', 'i', 'watch', 'in', \"it's\", 'wake', 'will', 'only', 'pale', 'in', 'comparison,', 'forget', 'your', 'michael', 'bay,', 'ridley', 'scott', 'slick', 'hollywood', 'cinematography,', 'europa', 'has', 'more', 'ethereal', 'beauty', 'than', 'anything', 'those', 'two', 'could', 'conjure', 'up', 'in', 'a', 'million', 'years.', 'now', \"i'd\", 'be', 'the', 'first', 'to', 'hail', 'lars', 'von', 'trier', 'a', 'genius', 'just', 'off', 'the', 'back', 'of', 'his', 'films', 'breaking', 'the', 'waves', 'and', 'dancer', 'in', 'the', 'dark,', 'but', 'this', 'is', 'stupid,', 'the', 'fact', 'that', 'europa', 'has', 'gone', 'un-noticed', 'by', 'film', 'experts', 'for', 'so', 'long', 'is', 'a', 'crime', 'against', 'cinema,', 'whilst', 'overrated', 'rubbish', 'like', 'crouching', 'tiger,', 'hidden', 'dragon', 'and', 'life', 'is', 'beautiful', 'clean', 'up', 'at', 'the', 'academy', 'awards', '(but', 'what', 'do', 'the', 'know)', 'europa', 'has', 'been', 'hidden', 'away,', 'absent', 'form', 'video', 'stores', 'and', '(until', 'recently)', 'any', 'british', 'tv', 'channels.', '<br', '/><br', '/>the', 'visuals', 'in', 'europa', 'are', 'not', 'mtv', 'gloss;', \"it's\", 'not', 'a', 'case', 'of', 'style', 'over', 'substance,', 'its', 'more', 'a', 'case', 'of', 'substance', 'dictating', 'style.', 'much', 'like', 'his', 'first', 'film', 'the', 'element', 'of', 'crime,', 'von', 'trier', 'uses', 'the', 'perspective', 'of', 'the', 'main', 'character', 'to', 'draw', 'us', 'into', 'his', 'world,', 'and', 'much', 'like', 'element,', 'the', 'film', 'begins', 'with', 'the', 'main', 'character', '(or', 'in', 'the', 'case', 'of', 'europa,', 'we', 'the', 'audience)', 'being', 'hypnotized.', 'as', 'we', 'move', 'down', 'the', 'tracks,', 'the', 'voice', 'of', 'the', 'narrator', '(max', 'von', 'sydow)', 'counts', 'us', 'down', 'into', 'a', 'deep', 'sleep,', 'until', 'we', 'awake', 'in', 'europa.', 'this', 'allows', 'von', 'trier', 'and', 'his', 'three', 'cinematographers', 'to', 'pay', 'with', 'the', 'conventions', 'of', 'time', 'and', 'imagery,', 'there', 'are', 'many', 'scenes', 'in', 'europa', 'when', 'a', 'character', 'in', 'the', 'background,', 'who', 'is', 'in', 'black', 'and', 'white,', 'will', 'interact', 'with', 'a', 'person', 'in', 'the', 'foreground', 'who', 'will', 'be', 'colour,', 'von', 'trier', 'is', 'trying', 'to', 'show', 'us', 'how', 'much', 'precedence', 'the', 'coloured', 'item', 'or', 'person', 'has', 'over', 'the', 'plot,', 'for', 'instance,', \"it's\", 'no', 'surprise', 'that', 'the', 'first', 'shot', 'of', 'leopold', 'kessler', '(jean-marc', 'barr)', 'is', 'in', 'colour,', 'since', 'he', 'is', 'the', 'only', 'character', \"who's\", 'actions', 'have', 'superiority', 'over', 'the', 'film.', '<br', '/><br', '/>the', 'performances', 'are', 'good,', 'they', 'may', 'not', 'be', 'on', 'par', 'with', 'performances', 'in', 'later', 'von', 'trier', 'films,', 'but', \"that's\", 'just', 'because', 'the', 'images', 'are', 'sometimes', 'so', 'distracting', 'that', 'you', \"don't\", 'really', 'pick', 'up', 'on', 'them', 'the', 'first', 'time', 'round.', 'but', 'i', 'would', 'like', 'to', 'point', 'out', 'the', 'fantastic', 'performance', 'of', 'jean-marc', 'barr', 'in', 'the', 'lead', 'role,', 'whose', 'blind', 'idealism', 'is', 'slowly', 'warn', 'down', 'by', 'the', 'two', 'opposing', 'sides,', 'until', 'he', 'erupts', 'in', 'the', 'films', 'final', 'act.', 'again,', 'muck', 'like', 'the', 'element', 'of', 'crime,', 'the', 'film', 'ends', 'with', 'our', 'hero', 'unable', 'to', 'wake', 'up', 'from', 'his', 'nightmare', 'state,', 'left', 'in', 'this', 'terrible', 'place,', 'with', 'only', 'the', 'continuing', 'narration', 'of', 'von', 'sydow', 'to', 'seal', 'his', 'fate.', 'europa', 'is', 'a', 'tremendous', 'film,', 'and', 'i', 'cant', 'help', 'thinking', 'what', 'a', 'shame', 'that', 'von', 'trier', 'has', 'abandoned', 'this', 'way', 'of', 'filming,', 'since', 'he', 'was', 'clearly', 'one', 'of', 'the', 'most', 'talented', 'visual', 'directors', 'working', 'at', 'that', 'time,', 'europa,', 'much', 'like', 'the', 'rest', 'of', 'his', 'cinematic', 'cannon', 'is', 'filled', 'with', 'a', 'wealth', 'of', 'iconic', 'scenes.', 'his', 'dedication', 'to', 'composition', 'and', 'mise-en-scene', 'is', 'unrivalled,', 'not', 'to', 'mention', 'his', 'use', 'of', 'sound', 'and', 'production', 'design.', 'but', 'since', 'his', 'no-frills', 'melodramas', 'turned', 'out', 'to', 'be', 'breaking', 'the', 'waves', 'and', 'dancer', 'in', 'the', 'dark', 'then', 'who', 'can', 'argue,', 'but', 'it', 'does', 'seems', 'like', 'a', 'waste', 'of', 'an', 'imaginative', 'talent.', '10/10'], 1)]\n"
     ]
    }
   ],
   "source": [
    "def data_preprocess(corpus):\n",
    "    data_set = []\n",
    "    for sentence, sentence_label in corpus:\n",
    "        #这里有一个小trick是把所有的句子转换为小写，从而减小词表的大小\n",
    "        #一般来说这样的做法有助于效果提升\n",
    "        sentence = sentence.strip().lower()\n",
    "        sentence = sentence.split(\" \")\n",
    "        \n",
    "        data_set.append((sentence, sentence_label))\n",
    "\n",
    "    return data_set\n",
    "\n",
    "train_corpus = data_preprocess(train_corpus)\n",
    "test_corpus = data_preprocess(test_corpus)\n",
    "print(train_corpus[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在经过切词后，需要构造一个词典，把每个词都转化成一个ID，以便于神经网络训练。代码如下：\n",
    "\n",
    "------\n",
    "**注意：**\n",
    "\n",
    "在代码中我们使用了一个特殊的单词\"[oov]\"（out-of-vocabulary），用于表示词表中没有覆盖到的词。之所以使用\"[oov]\"这个符号，是为了处理某一些词，再测试数据中有，但训练数据没有的现象。\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "there are totoally 252173 different words in the corpus\n",
      "word [oov], its id 0, its word freq 10000000000\n",
      "word [pad], its id 1, its word freq 10000000000\n",
      "word the, its id 2, its word freq 322174\n",
      "word a, its id 3, its word freq 159949\n",
      "word and, its id 4, its word freq 158556\n",
      "word of, its id 5, its word freq 144459\n",
      "word to, its id 6, its word freq 133965\n",
      "word is, its id 7, its word freq 104170\n",
      "word in, its id 8, its word freq 90521\n",
      "word i, its id 9, its word freq 70477\n",
      "word this, its id 10, its word freq 69711\n",
      "word that, its id 11, its word freq 66288\n",
      "word it, its id 12, its word freq 65490\n",
      "word /><br, its id 13, its word freq 50935\n",
      "word was, its id 14, its word freq 47023\n",
      "word as, its id 15, its word freq 45098\n",
      "word for, its id 16, its word freq 42840\n",
      "word with, its id 17, its word freq 42725\n",
      "word but, its id 18, its word freq 39757\n",
      "word on, its id 19, its word freq 31618\n",
      "word movie, its id 20, its word freq 30885\n",
      "word his, its id 21, its word freq 29058\n",
      "word are, its id 22, its word freq 28742\n",
      "word not, its id 23, its word freq 28591\n",
      "word film, its id 24, its word freq 27774\n",
      "word you, its id 25, its word freq 27558\n",
      "word have, its id 26, its word freq 27343\n",
      "word he, its id 27, its word freq 26175\n",
      "word be, its id 28, its word freq 25690\n",
      "word at, its id 29, its word freq 22730\n",
      "word one, its id 30, its word freq 22478\n",
      "word by, its id 31, its word freq 21974\n",
      "word an, its id 32, its word freq 21239\n",
      "word they, its id 33, its word freq 20623\n",
      "word from, its id 34, its word freq 19933\n",
      "word all, its id 35, its word freq 19738\n",
      "word who, its id 36, its word freq 19407\n",
      "word like, its id 37, its word freq 18779\n",
      "word so, its id 38, its word freq 18095\n",
      "word just, its id 39, its word freq 17308\n",
      "word or, its id 40, its word freq 16765\n",
      "word has, its id 41, its word freq 16570\n",
      "word her, its id 42, its word freq 16538\n",
      "word about, its id 43, its word freq 16486\n",
      "word it's, its id 44, its word freq 15969\n",
      "word some, its id 45, its word freq 15280\n",
      "word if, its id 46, its word freq 15184\n",
      "word out, its id 47, its word freq 14510\n",
      "word what, its id 48, its word freq 14053\n",
      "word very, its id 49, its word freq 13633\n"
     ]
    }
   ],
   "source": [
    "#构造词典，统计每个词的频率，并根据频率将每个词转换为一个整数id\n",
    "def build_dict(corpus):\n",
    "    word_freq_dict = dict()\n",
    "    for sentence, _ in corpus:\n",
    "        for word in sentence:\n",
    "            if word not in word_freq_dict:\n",
    "                word_freq_dict[word] = 0\n",
    "            word_freq_dict[word] += 1\n",
    "\n",
    "    word_freq_dict = sorted(word_freq_dict.items(), key = lambda x:x[1], reverse = True)\n",
    "    \n",
    "    word2id_dict = dict()\n",
    "    word2id_freq = dict()\n",
    "\n",
    "    #一般来说，我们把oov和pad放在词典前面，给他们一个比较小的id，这样比较方便记忆，并且易于后续扩展词表\n",
    "    word2id_dict['[oov]'] = 0\n",
    "    word2id_freq[0] = 1e10\n",
    "\n",
    "    word2id_dict['[pad]'] = 1\n",
    "    word2id_freq[1] = 1e10\n",
    "\n",
    "    for word, freq in word_freq_dict:\n",
    "        word2id_dict[word] = len(word2id_dict)\n",
    "        word2id_freq[word2id_dict[word]] = freq\n",
    "\n",
    "    return word2id_freq, word2id_dict\n",
    "\n",
    "word2id_freq, word2id_dict = build_dict(train_corpus)\n",
    "vocab_size = len(word2id_freq)\n",
    "print(\"there are totoally %d different words in the corpus\" % vocab_size)\n",
    "for _, (word, word_id) in zip(range(50), word2id_dict.items()):\n",
    "    print(\"word %s, its id %d, its word freq %d\" % (word, word_id, word2id_freq[word_id]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在完成word2id词典假设之后，我们还需要进一步处理原始语料，把语料中的所有句子都处理成ID序列，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000 tokens in the corpus\n",
      "[([2, 956, 9, 308, 16, 10, 20, 66, 173, 518, 38, 9, 761, 6, 896, 12, 653, 97, 31833, 10, 7, 167, 20, 43, 3, 1343, 47, 16, 1397, 375, 3, 587, 2635, 2, 75, 7, 2353, 1594, 103, 574, 35042, 17, 30, 182, 4827, 1945, 3, 67291, 5, 9, 118, 48, 25, 113, 224, 9922, 4, 759, 3, 2503, 11223, 5, 615, 480, 618, 70, 203, 3996, 313, 302, 26419, 52, 7, 164, 154, 6, 10, 728, 4, 20123, 2, 2570, 190, 2314, 38, 4239, 5, 262, 55, 543, 10, 293, 10, 20, 2994, 6, 4, 34, 2, 584, 104, 1833, 4, 295, 9, 95, 318, 5, 12, 35, 9, 1735, 12, 126, 147, 91, 76, 1736, 121, 135, 34, 9573, 6, 3369, 522, 8, 2, 13595, 5, 32, 6033, 1681, 5, 646, 33, 22, 35, 111, 3114, 4929, 70, 2, 6280, 5462, 11204, 2, 39597, 2838, 2, 26376, 2648, 3770, 2, 433, 68444, 2, 3494, 1842, 2, 4430, 2198, 27758, 2, 216246, 314, 1842, 3, 19928, 328, 216247, 4, 9, 147, 456, 43, 92, 5, 429, 382, 3, 138, 906, 90, 26, 95, 2, 20, 1090, 52, 66, 45, 518, 159, 8, 204, 18, 726, 10, 20, 14, 3, 2795, 9, 128, 81, 710, 3, 720, 34877, 216248, 11, 54, 26, 71, 1240, 6, 21988, 109, 13, 255, 20, 205, 2, 238, 2376, 213, 41, 6, 1749, 31, 3575, 18, 12, 7, 126, 173, 598, 46, 25, 39, 171, 6, 67, 45, 1115, 1385, 8, 2, 2650, 40, 39, 171, 6, 576, 47, 1849, 29, 45, 127, 2, 426, 590, 40, 39, 171, 6, 10502, 8800, 38880, 29, 3, 20, 2046, 40, 39, 171, 6, 26, 167, 2376, 213, 844, 65, 111, 39106, 50, 277, 1638, 25, 79, 100, 786, 119, 70, 10, 20, 7, 16, 933, 109, 13, 9692, 3152, 3, 518, 131, 166, 7723], 0), ([57, 378, 4, 9, 199, 11, 17, 10, 212, 4, 1421, 2, 20, 54, 26, 6, 28, 29, 219, 267, 2150, 69, 66, 1539, 8, 634, 69, 442, 65, 19, 12, 94, 4640, 1481, 2, 321, 11, 25003, 185, 4, 1339, 22, 632, 18, 1247, 17, 299, 3519, 1224, 16, 493, 1240, 6, 73, 584, 322, 261, 6, 250, 10, 7, 2998, 50, 60, 30, 2414, 17, 3, 632, 4975, 52, 7, 1834, 1235, 187, 7877, 4, 4920, 4, 6, 10805, 3, 868, 10975, 2, 321, 11, 4014, 4, 7877, 1118, 12587, 59, 120, 16, 30, 167, 191, 92, 307, 7, 9687, 52, 7, 60, 307, 6, 456, 43, 92, 5, 2, 646, 63, 7, 144, 69, 3890, 8, 2, 16873, 2804, 2045, 9, 236, 12, 261, 6, 250, 11, 25692, 533, 471, 15, 603, 5, 2, 184, 5241, 17, 2, 5673, 82585, 91915, 13859, 38, 544, 111, 390, 4, 1803, 10, 457], 0), ([10, 7, 2, 381, 5, 3, 154, 4, 245, 374, 2686, 1284, 12, 41, 3, 15514, 6, 102524, 5, 223, 17, 4151, 4, 327, 102525, 102526, 13, 93, 121, 311, 45, 51, 84, 6, 2164, 18, 33, 26, 4997, 30, 174, 11, 7, 925, 1872, 1609, 35, 374, 2686, 260, 23892, 7, 11, 33, 1930, 130, 3, 8775, 541, 5, 2, 25587, 187, 404, 4, 418, 8, 2, 910, 50, 33, 2858, 3, 49, 1117, 541, 43, 292, 1657, 102527, 41, 2306, 2246, 32573, 70214, 13, 768, 2, 618, 121, 198, 22, 3818, 55501, 32560, 36, 7, 20064, 4, 23885, 2479, 35923, 36, 7, 8145, 35923, 7, 168, 14975, 4, 32560, 7, 95, 6, 28, 3, 102528, 102529, 13, 768, 45, 5, 2, 1187, 6023, 223, 3, 218, 104, 299, 762, 3599, 6, 2, 51, 1902, 223, 5, 2, 102530, 18, 2088, 245, 46525, 13, 768, 3, 49, 56, 2316, 192, 16, 3, 49, 56, 1284], 1), ([10, 4259, 15, 30, 5, 2, 238, 123, 190, 119, 8, 1043, 1832, 5934, 4, 69316, 2, 132, 7, 157, 11260, 262, 12732, 295, 3, 518, 57310, 48, 698, 6, 7797, 2, 244, 7, 518, 18, 302, 22863, 2, 91, 65, 7, 38, 97, 44, 267, 33783, 9, 1344, 36, 125, 55, 1217, 43, 2, 3028, 8, 3, 2455, 904, 139, 277, 128, 337, 2, 91, 65, 9543, 4, 2, 93884, 11, 298, 23, 356, 6, 157, 67, 112, 2, 336, 7, 3, 2505, 2474, 70, 360, 2, 478, 5, 13985, 96, 33, 1735, 12, 14, 35, 3, 241237, 13, 2585, 393, 111, 84, 40, 111, 1698, 298, 138, 137, 39, 4679, 72, 1060, 16, 395, 3851], 0), ([7207, 7, 3, 88, 20, 17, 3, 88, 75, 43, 3, 88, 118507, 2132, 75770, 10, 7, 5, 532, 2, 75, 5, 30, 5, 5391, 88, 1307, 16768, 4, 3, 848, 95, 1163, 6, 87, 34, 2, 8620, 474, 5, 57, 8380, 192, 52, 7, 3, 3087, 2228, 5633, 186, 28, 11200, 8928, 15872, 3118, 1017, 8, 2, 24, 15, 21, 21847, 6872, 6, 2, 2449, 1307, 1864, 19, 7598, 2, 118508, 1593, 113, 23, 2986, 2, 7311, 8624, 1541, 102, 75787, 33, 39, 6970, 37662, 3196, 14, 8, 21, 3118, 6, 16685, 691, 21, 37193, 31, 2703, 118509, 8, 11811, 16, 21, 7083, 118510, 25520, 103, 113, 23, 67, 1054, 6, 118511, 6525, 8862, 7, 118512, 15, 2, 959, 36, 37139, 27, 54, 1066, 6, 2, 28651, 8, 234, 408, 7858, 2, 24, 1114, 1048, 4, 660, 17, 2, 5385, 21, 256, 4, 21, 4871, 34, 2, 505, 5, 2, 332, 234, 408, 6, 2, 172, 5, 21, 3218, 2749, 10, 24, 54, 28, 5, 76, 2729, 7009, 6, 30, 1163, 17, 193, 15392, 2782, 4, 2, 3804, 2369, 3852, 23170, 1128, 5, 10, 88, 160, 36, 5762, 2, 4247, 22727, 5, 18490, 4, 13619, 4, 36, 1777, 408, 15, 58, 3, 1963, 64, 7, 3, 1055, 30, 6102, 4009, 408, 52, 7, 60, 9139, 16, 75788], 1)]\n",
      "[([868, 2313, 29392, 0, 442, 3, 76, 138, 20739, 5, 2, 502, 131, 1724, 68, 9, 9002, 48, 9, 2606, 636, 7, 11, 12, 14, 39, 38, 2338, 8, 158, 318, 70, 17859, 14000, 0, 70, 39, 0, 13, 558, 25, 37, 123, 11, 22, 400, 6, 166, 5447, 4, 81, 26, 869, 1976, 6, 63, 11, 1140, 7, 92312, 9, 98, 25, 74, 28, 1182, 6, 26, 119, 10, 3195, 4, 1865, 210, 5, 13350, 13, 2435, 3, 3873, 5, 815, 6, 3669, 428, 183, 12, 43, 32, 0, 2, 58, 307, 9, 4506, 285, 34, 1802, 7, 11, 12, 7, 3, 1519, 6128, 46, 25, 22, 8, 2, 1519, 16, 3, 61, 21792, 49, 714, 151, 96, 44, 3, 1317, 9, 355, 98, 44, 3, 39542, 18, 603, 5, 188, 64, 28, 8, 11, 1519, 35, 2, 390, 767, 1671, 0], 1), ([1292, 90413, 11644, 110, 10, 2004, 80, 151, 0, 29, 45, 1912, 44, 3, 138, 151, 18, 10, 30, 205, 97, 39, 12004, 13, 742, 1272, 7050, 249, 5, 797, 34, 32, 1382, 10638, 16256, 90851, 21795, 42797, 4, 2, 1128, 5, 8494, 454, 841, 91, 10, 32, 2224, 4, 26441, 4321, 24, 987, 2531, 35, 5, 2, 1116, 121, 504, 2, 68873, 5, 2, 1756, 1987, 17, 59, 5583, 49133, 30, 828, 7, 96790, 4, 7250, 3, 926, 5, 4004, 0, 31, 647, 35, 5, 21, 1448, 19126, 84, 4, 1387, 6, 21, 1768, 0, 0, 21, 378, 7, 3015, 4, 19969, 16, 14407, 4, 15421, 63, 53, 853, 554, 17, 42, 31505, 2741, 37003, 36, 239, 6, 422, 8, 59, 412, 17, 42, 15064, 5542, 10, 15064, 828, 7, 8971, 8, 120, 17, 21, 1687, 7032, 18, 14, 923, 72, 1756, 17, 3, 0, 6, 13450, 28289, 33, 58, 26, 454, 295, 50, 21, 6448, 2741, 378, 1900, 42, 251602, 13, 742, 7812, 517, 8, 2, 412, 4, 27, 853, 2594, 2, 1093, 11, 2, 103, 418, 22, 10682, 167, 2743, 122, 7, 2, 3937, 33049, 36, 7, 1965, 6, 1247, 40, 456, 16, 907, 610, 6, 3, 31531, 248, 53, 988, 3, 16162, 6675, 6, 6534, 42, 676, 15, 107, 15, 42, 14154, 17, 2, 256, 10056, 53, 1620, 115, 118, 140, 42, 6675, 40, 31, 17608, 42, 7054, 11, 53, 697, 0, 777, 162, 19, 8, 2, 412, 4, 79, 76, 53, 0, 13, 537, 2, 656, 2, 1205, 43, 1602, 239, 47, 4, 2, 103, 618, 2545, 172, 65, 578, 285, 1374, 712, 23, 179, 52, 7, 32, 869, 150, 187, 2, 0, 828, 4, 21, 12564, 812, 2959, 42, 202678, 4145, 19, 1340, 4, 29, 82, 69, 98, 53, 7, 162, 6, 5186, 248, 69, 67, 2, 103, 418, 2362, 8, 2, 49, 224, 150, 5, 2, 708, 13, 93, 3845, 5, 10, 24, 6216, 42, 55389, 25766, 18, 53, 287, 188, 2485, 396, 5133, 36, 22, 2768, 31, 59, 1701, 4, 24850, 69, 208, 6, 61, 456, 43, 2, 121, 4, 69, 86, 67, 115, 15, 10190, 233, 19, 3552, 188, 17, 59, 10636, 57263, 35711, 9493, 4, 8425], 1), ([10, 7, 57, 82, 216402, 0, 139, 9, 200, 2, 24, 19, 280, 8, 83, 6894, 324, 17, 83, 0, 122, 1394, 15, 0, 9, 81, 1271, 11, 12, 7, 20940, 36, 40892, 2, 18277, 4302, 31, 1340, 8, 2, 24, 4, 23, 0, 266, 48, 9, 403, 43, 2, 24, 14, 83, 1088, 31, 8022, 216403, 23, 42, 3123, 2, 646, 196, 4, 7665, 66, 20084, 4, 401, 155068, 854, 2, 0, 37, 2, 172, 5, 0, 11022, 33942, 61182, 33, 22, 2955, 418, 3483, 31, 3801, 3701, 4, 30, 3801, 160, 0, 175, 6, 208, 1989, 8, 2, 224, 339, 69, 67, 5, 0, 21, 58262, 542, 0, 27, 175, 6, 449, 180, 19, 3, 910, 5816, 675, 244102, 4, 0, 6, 8826, 109, 13, 80128, 0, 175, 6, 41315, 15, 3, 201, 31541, 1025, 6, 78, 1382, 2765, 627, 53, 54, 268, 6, 28, 150953, 15, 53, 486, 316, 42, 272, 6, 135, 675, 2, 73962, 10695, 256, 53, 7, 5535, 557, 5535, 3, 3067, 5, 2, 1687, 0, 8, 2, 994, 23108, 5, 11517, 53, 55, 46471, 2, 24, 6, 42, 542, 4, 778, 1472, 42, 0, 266, 42, 20940, 0, 5, 149611, 734, 17, 193, 42, 1018, 8, 3, 47624, 12488, 1633, 42, 6, 0, 84558, 0, 0, 0, 10000, 4, 53, 7, 43092, 31, 124, 0, 42, 3863, 2427, 411, 921, 4, 4209, 0, 191428, 24, 5, 103, 20338, 2391, 8, 32, 1382, 9789, 6237, 186, 28, 56515, 6, 7465, 18, 1400, 2200, 2, 0, 5, 9789, 9049, 11, 114, 23, 268, 6, 16896, 16, 143, 138, 68, 83, 5450, 5692, 8, 3, 1806, 994, 7079, 0, 0, 4, 0, 129, 26, 2957, 5328, 1521, 8, 4817, 18, 113, 23, 26, 2, 8244, 11, 3483, 10, 24, 4, 1934, 26, 23, 71, 119, 31, 3, 2205, 2614, 5, 17554, 0, 13, 80128, 97518, 8022, 0, 1722, 44063, 4, 1722, 125301, 22, 3917, 18, 23, 7588, 8022, 54565, 7, 3, 1039, 633, 36, 442, 1096, 423, 441, 56, 998, 0, 216054, 0, 241343, 0, 0, 0, 0, 3, 4850, 4134, 7109, 8, 10, 139, 8022, 26168, 0, 610, 6, 42, 387, 1558, 228, 68, 42, 132, 78795, 35, 8, 364, 2, 502, 2434, 1252, 8, 2, 3458, 5, 2, 1088, 63, 7, 801, 953, 8, 1403, 5, 1902, 1811, 9, 226, 274, 8022, 0, 64, 53472, 42, 553, 2447, 8, 42, 910, 59369], 1), ([10, 14, 3, 88, 24, 8, 158, 318, 5, 2, 6064, 12, 14409, 2, 961, 5, 0, 8, 3, 1333, 11, 7, 168, 44886, 5, 92, 46459, 34, 2, 29343, 12, 104671, 3, 88, 100, 1382, 14399, 11, 100, 153, 12790, 322, 130, 15, 2, 32900, 5, 16879, 31, 2751, 2, 4031, 5, 3564, 4, 3361, 1768, 60550, 59993, 2, 132989, 5, 418, 8, 1403, 5, 59, 0, 92282, 2, 37268, 8194, 5, 5707, 4, 38, 774, 29, 2, 657, 5, 2, 24, 7, 3, 1465, 120, 75, 11, 254, 675, 130, 8483, 5, 2, 1333, 63, 2, 103, 4213, 153, 2392, 2, 24, 7, 10064, 4, 19067, 302, 1157, 34, 505, 6, 656, 1030, 78, 3184, 7144, 2, 258, 8293, 5, 2, 24, 15, 9, 200, 12, 14, 1263, 11, 2, 103, 2545, 268, 1422, 6, 30, 167, 23, 2679, 31, 3, 1393, 15820, 16, 233, 80, 15, 76, 15, 2, 231, 11, 33, 22, 1499, 8, 26704, 16879, 17, 60, 2209, 4, 60, 80763, 10, 186, 304, 3, 182, 8, 2, 810, 8706, 5, 2, 646, 18, 78, 89, 1499, 8, 2, 163, 1173, 74, 23, 33671, 0, 12, 175, 827, 34, 2, 505, 5, 2, 24, 11, 2, 103, 121, 22, 168, 16049, 322, 50, 20940, 114, 42, 150, 29, 2, 172, 5, 2, 20, 17, 0, 53, 146, 12, 168, 827, 11, 41096, 1941, 53, 14, 30531, 4, 2, 13046, 14, 11, 46, 27, 62, 9486, 767, 27, 90, 26, 15563, 42, 168, 17836, 4, 81, 50, 0, 401, 49, 1084, 50, 42, 828, 401, 6, 23, 37, 717, 130, 1718, 85, 23, 453, 89, 72, 31404, 322, 33, 186, 2822, 5803, 8, 498, 8, 2, 163, 14511, 18, 0, 24, 12, 7, 23, 29, 35, 95, 827, 11, 33, 22, 17280, 34, 2, 505, 322, 168, 2, 8308, 45, 89, 22, 0, 12, 7, 2698, 18, 78, 2196, 6, 28, 452, 0, 40, 40327, 8, 2, 529, 5, 2, 2391, 8, 2, 151, 193, 62, 22137, 17802, 16, 7765, 421, 421, 421, 46, 10, 62, 23, 71, 2, 1707, 54, 33, 26, 3186, 236, 2, 311, 6, 2790, 59, 1941, 8, 3, 667, 11, 33, 186, 26, 1181, 23, 26, 0, 2, 24, 7891, 464, 2, 80, 2870, 7, 2, 13748, 5, 2, 121, 421, 421, 421, 2, 1652, 0, 4, 20940, 268, 3013, 4182, 6, 1787, 4, 15326, 31987, 3, 24, 8, 1416, 5236, 3, 1099, 667, 187, 3, 160, 699, 66871, 4, 167, 699, 0, 31, 695, 1652, 3409, 17, 898, 0, 68086, 2, 24, 7, 566, 221, 6, 814, 19, 83, 209, 4, 676, 60, 130, 7113, 8, 57, 3462, 29, 92, 8513, 2, 8293, 85, 23, 180, 76, 285, 34, 2, 818, 5, 2, 117, 12, 7, 1265, 3, 49, 1465, 4, 1057, 75, 322, 2, 1385, 4, 121, 74, 758, 17, 25, 3, 229, 84, 94, 25, 536, 2, 10270], 1), ([3, 6550, 7527, 151, 17, 4097, 590, 3932, 2488, 4, 60436, 13, 742, 45757, 554, 1116, 161, 23, 48, 53, 5868, 4, 458, 6, 1083, 42, 5707, 6, 907, 4472, 42, 5707, 6, 3814, 373, 17, 42, 1026, 0, 53, 146, 45, 7399, 88965, 34, 4507, 841, 4, 0], 1)]\n"
     ]
    }
   ],
   "source": [
    "#把语料转换为id序列\n",
    "def convert_corpus_to_id(corpus, word2id_dict):\n",
    "    data_set = []\n",
    "    for sentence, sentence_label in corpus:\n",
    "        #将句子中的词逐个替换成id，如果句子中的词不在词表内，则替换成oov\n",
    "        #这里需要注意，一般来说我们可能需要查看一下test-set中，句子oov的比例，\n",
    "        #如果存在过多oov的情况，那就说明我们的训练数据不足或者切分存在巨大偏差，需要调整\n",
    "        sentence = [word2id_dict[word] if word in word2id_dict \\\n",
    "                    else word2id_dict['[oov]'] for word in sentence]    \n",
    "        data_set.append((sentence, sentence_label))\n",
    "    return data_set\n",
    "\n",
    "train_corpus = convert_corpus_to_id(train_corpus, word2id_dict)\n",
    "test_corpus = convert_corpus_to_id(test_corpus, word2id_dict)\n",
    "print(\"%d tokens in the corpus\" % len(train_corpus))\n",
    "print(train_corpus[:5])\n",
    "print(test_corpus[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们就可以开始把原始预料中的每个句子通过截断和填充，转换成一个固定长度的句子，并将所有数据整理成mini-batch，用于训练模型，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(array([[[ 35278],\n",
      "        [     7],\n",
      "        [     3],\n",
      "        [    49],\n",
      "        [  3171],\n",
      "        [   391],\n",
      "        [     5],\n",
      "        [    24],\n",
      "        [157929],\n",
      "        [     6],\n",
      "        [    28],\n",
      "        [157930],\n",
      "        [    12],\n",
      "        [  1457],\n",
      "        [    87],\n",
      "        [     5],\n",
      "        [     2],\n",
      "        [  1433],\n",
      "        [157931],\n",
      "        [     8],\n",
      "        [     3],\n",
      "        [   108],\n",
      "        [  6373],\n",
      "        [ 17573],\n",
      "        [   781],\n",
      "        [  3007],\n",
      "        [     4],\n",
      "        [  2548],\n",
      "        [  3110],\n",
      "        [    19]],\n",
      "\n",
      "       [[  8070],\n",
      "        [    24],\n",
      "        [    43],\n",
      "        [     3],\n",
      "        [    56],\n",
      "        [   275],\n",
      "        [    18],\n",
      "        [  8070],\n",
      "        [   351],\n",
      "        [    36],\n",
      "        [  1119],\n",
      "        [     6],\n",
      "        [   422],\n",
      "        [    59],\n",
      "        [  1179],\n",
      "        [     4],\n",
      "        [   180],\n",
      "        [    89],\n",
      "        [    19],\n",
      "        [  9311],\n",
      "        [ 68731],\n",
      "        [     2],\n",
      "        [  2214],\n",
      "        [   202],\n",
      "        [  4006],\n",
      "        [    91],\n",
      "        [     2],\n",
      "        [   539],\n",
      "        [  1291],\n",
      "        [     5]],\n",
      "\n",
      "       [[     9],\n",
      "        [   262],\n",
      "        [   945],\n",
      "        [    10],\n",
      "        [     6],\n",
      "        [    28],\n",
      "        [     3],\n",
      "        [    88],\n",
      "        [   155],\n",
      "        [    18],\n",
      "        [  1098],\n",
      "        [    14],\n",
      "        [     9],\n",
      "        [   945],\n",
      "        [    12],\n",
      "        [     6],\n",
      "        [    28],\n",
      "        [    38],\n",
      "        [  1534],\n",
      "        [     9],\n",
      "        [  1777],\n",
      "        [     2],\n",
      "        [   542],\n",
      "        [   122],\n",
      "        [    38],\n",
      "        [    76],\n",
      "        [     9],\n",
      "        [    62],\n",
      "        [     6],\n",
      "        [   453]]]), array([[1],\n",
      "       [0],\n",
      "       [0]]))\n",
      "(array([[[   10],\n",
      "        [   20],\n",
      "        [    7],\n",
      "        [    3],\n",
      "        [  169],\n",
      "        [   37],\n",
      "        [    2],\n",
      "        [   20],\n",
      "        [44557],\n",
      "        [  557],\n",
      "        [   17],\n",
      "        [68622],\n",
      "        [  132],\n",
      "        [    4],\n",
      "        [   23],\n",
      "        [   76],\n",
      "        [ 4401],\n",
      "        [    2],\n",
      "        [  819],\n",
      "        [  824],\n",
      "        [   22],\n",
      "        [ 1002],\n",
      "        [   18],\n",
      "        [   25],\n",
      "        [   86],\n",
      "        [   61],\n",
      "        [  223],\n",
      "        [  237],\n",
      "        [   16],\n",
      "        [    2]],\n",
      "\n",
      "       [[    9],\n",
      "        [  486],\n",
      "        [  250],\n",
      "        [   79],\n",
      "        [ 1105],\n",
      "        [   10],\n",
      "        [  134],\n",
      "        [  701],\n",
      "        [    9],\n",
      "        [ 1081],\n",
      "        [  608],\n",
      "        [   32],\n",
      "        [ 7870],\n",
      "        [ 1095],\n",
      "        [ 3240],\n",
      "        [    9],\n",
      "        [   26],\n",
      "        [  308],\n",
      "        [ 3382],\n",
      "        [ 1095],\n",
      "        [ 4248],\n",
      "        [    4],\n",
      "        [   26],\n",
      "        [  447],\n",
      "        [  100],\n",
      "        [ 1095],\n",
      "        [  123],\n",
      "        [    4],\n",
      "        [  280],\n",
      "        [ 3536]],\n",
      "\n",
      "       [[  190],\n",
      "        [  308],\n",
      "        [  100],\n",
      "        [ 1550],\n",
      "        [  956],\n",
      "        [    5],\n",
      "        [   10],\n",
      "        [   20],\n",
      "        [    4],\n",
      "        [  449],\n",
      "        [  176],\n",
      "        [    3],\n",
      "        [  674],\n",
      "        [    6],\n",
      "        [   67],\n",
      "        [   12],\n",
      "        [   19],\n",
      "        [ 1886],\n",
      "        [    6],\n",
      "        [   28],\n",
      "        [ 1728],\n",
      "        [    9],\n",
      "        [   61],\n",
      "        [   86],\n",
      "        [  118],\n",
      "        [   48],\n",
      "        [    2],\n",
      "        [  461],\n",
      "        [   17],\n",
      "        [   12]]]), array([[0],\n",
      "       [0],\n",
      "       [1]]))\n",
      "(array([[[    10],\n",
      "        [    14],\n",
      "        [   266],\n",
      "        [   167],\n",
      "        [   196],\n",
      "        [   387],\n",
      "        [  8335],\n",
      "        [    16],\n",
      "        [     3],\n",
      "        [   188],\n",
      "        [   280],\n",
      "        [   134],\n",
      "        [    34],\n",
      "        [     2],\n",
      "        [  7819],\n",
      "        [    12],\n",
      "        [     7],\n",
      "        [  1383],\n",
      "        [   221],\n",
      "        [    18],\n",
      "        [    14],\n",
      "        [    49],\n",
      "        [    76],\n",
      "        [     6],\n",
      "        [  9785],\n",
      "        [  1246],\n",
      "        [  9028],\n",
      "        [  6187],\n",
      "        [    19],\n",
      "        [  1040]],\n",
      "\n",
      "       [[    30],\n",
      "        [     5],\n",
      "        [     2],\n",
      "        [   116],\n",
      "        [   465],\n",
      "        [   489],\n",
      "        [   123],\n",
      "        [    34],\n",
      "        [160629],\n",
      "        [     7],\n",
      "        [    10],\n",
      "        [     7],\n",
      "        [     2],\n",
      "        [   513],\n",
      "        [   589],\n",
      "        [     5],\n",
      "        [   154],\n",
      "        [ 46915],\n",
      "        [    46],\n",
      "        [    25],\n",
      "        [   250],\n",
      "        [     8],\n",
      "        [ 61781],\n",
      "        [   785],\n",
      "        [    86],\n",
      "        [   110],\n",
      "        [    10],\n",
      "        [  1545],\n",
      "        [    12],\n",
      "        [  4661]],\n",
      "\n",
      "       [[    52],\n",
      "        [    22],\n",
      "        [    49],\n",
      "        [   156],\n",
      "        [   123],\n",
      "        [    11],\n",
      "        [    22],\n",
      "        [    38],\n",
      "        [   235],\n",
      "        [    15],\n",
      "        [    10],\n",
      "        [   457],\n",
      "        [     9],\n",
      "        [    14],\n",
      "        [  2463],\n",
      "        [   221],\n",
      "        [     6],\n",
      "        [   110],\n",
      "        [    10],\n",
      "        [    20],\n",
      "        [    29],\n",
      "        [     3],\n",
      "        [   919],\n",
      "        [ 82968],\n",
      "        [    16],\n",
      "        [    20],\n",
      "        [  6944],\n",
      "        [    37],\n",
      "        [   493],\n",
      "        [    38]]]), array([[0],\n",
      "       [1],\n",
      "       [1]]))\n",
      "(array([[[    25],\n",
      "        [    86],\n",
      "        [   520],\n",
      "        [    76],\n",
      "        [    34],\n",
      "        [     3],\n",
      "        [ 26959],\n",
      "        [  3023],\n",
      "        [     4],\n",
      "        [    17],\n",
      "        [  1272],\n",
      "        [222833],\n",
      "        [    34],\n",
      "        [  4816],\n",
      "        [  6812],\n",
      "        [ 46313],\n",
      "        [     4],\n",
      "        [     3],\n",
      "        [   156],\n",
      "        [    31],\n",
      "        [  2388],\n",
      "        [  5461],\n",
      "        [  3502],\n",
      "        [    40],\n",
      "        [ 25485],\n",
      "        [222834],\n",
      "        [   549],\n",
      "        [    48],\n",
      "        [    25],\n",
      "        [ 60425]],\n",
      "\n",
      "       [[    32],\n",
      "        [   334],\n",
      "        [  1086],\n",
      "        [   732],\n",
      "        [ 10164],\n",
      "        [    17],\n",
      "        [  7843],\n",
      "        [  3063],\n",
      "        [   354],\n",
      "        [   375],\n",
      "        [    42],\n",
      "        [  1085],\n",
      "        [  2600],\n",
      "        [ 56881],\n",
      "        [  7385],\n",
      "        [     8],\n",
      "        [     3],\n",
      "        [   263],\n",
      "        [    11],\n",
      "        [ 11559],\n",
      "        [  8325],\n",
      "        [175903],\n",
      "        [   599],\n",
      "        [  1790],\n",
      "        [   189],\n",
      "        [  4288],\n",
      "        [    77],\n",
      "        [  7843],\n",
      "        [     7],\n",
      "        [  3282]],\n",
      "\n",
      "       [[    50],\n",
      "        [    12],\n",
      "        [   239],\n",
      "        [     6],\n",
      "        [   123],\n",
      "        [     9],\n",
      "        [    64],\n",
      "        [    28],\n",
      "        [   173],\n",
      "        [ 87864],\n",
      "        [     4],\n",
      "        [   625],\n",
      "        [  4214],\n",
      "        [    43],\n",
      "        [   237],\n",
      "        [     4],\n",
      "        [   292],\n",
      "        [    11],\n",
      "        [     7],\n",
      "        [   247],\n",
      "        [  1539],\n",
      "        [   133],\n",
      "        [   158],\n",
      "        [    20],\n",
      "        [    41],\n",
      "        [    83],\n",
      "        [  5947],\n",
      "        [     2],\n",
      "        [   394],\n",
      "        [ 14287]]]), array([[0],\n",
      "       [1],\n",
      "       [1]]))\n",
      "(array([[[     9],\n",
      "        [   236],\n",
      "        [  3835],\n",
      "        [ 69595],\n",
      "        [     6],\n",
      "        [    28],\n",
      "        [     3],\n",
      "        [  3967],\n",
      "        [   151],\n",
      "        [    32],\n",
      "        [  3009],\n",
      "        [166749],\n",
      "        [    13],\n",
      "        [ 16412],\n",
      "        [    29],\n",
      "        [   104],\n",
      "        [ 88914],\n",
      "        [    36],\n",
      "        [   711],\n",
      "        [    26],\n",
      "        [     2],\n",
      "        [   246],\n",
      "        [ 31998],\n",
      "        [166750],\n",
      "        [    13],\n",
      "        [ 85043],\n",
      "        [     4],\n",
      "        [   422],\n",
      "        [     8],\n",
      "        [     3]],\n",
      "\n",
      "       [[    10],\n",
      "        [    24],\n",
      "        [     7],\n",
      "        [   104],\n",
      "        [200225],\n",
      "        [    44],\n",
      "        [     3],\n",
      "        [   925],\n",
      "        [  1973],\n",
      "        [    24],\n",
      "        [   729],\n",
      "        [200226],\n",
      "        [    63],\n",
      "        [  7929],\n",
      "        [    12],\n",
      "        [ 24659],\n",
      "        [    18],\n",
      "        [     2],\n",
      "        [   504],\n",
      "        [     5],\n",
      "        [     3],\n",
      "        [   518],\n",
      "        [   212],\n",
      "        [     4],\n",
      "        [     3],\n",
      "        [  8192],\n",
      "        [    47],\n",
      "        [   131],\n",
      "        [  6361],\n",
      "        [    12]],\n",
      "\n",
      "       [[     2],\n",
      "        [ 11230],\n",
      "        [   174],\n",
      "        [    43],\n",
      "        [    10],\n",
      "        [    24],\n",
      "        [     7],\n",
      "        [    11],\n",
      "        [    58],\n",
      "        [  1447],\n",
      "        [    89],\n",
      "        [  4043],\n",
      "        [     6],\n",
      "        [   536],\n",
      "        [     3],\n",
      "        [   882],\n",
      "        [     5],\n",
      "        [    12],\n",
      "        [     4],\n",
      "        [  5611],\n",
      "        [   361],\n",
      "        [    12],\n",
      "        [  3944],\n",
      "        [  1145],\n",
      "        [     3],\n",
      "        [   999],\n",
      "        [    19],\n",
      "        [     2],\n",
      "        [   889],\n",
      "        [116480]]]), array([[1],\n",
      "       [0],\n",
      "       [1]]))\n",
      "(array([[[     9],\n",
      "        [    26],\n",
      "        [    71],\n",
      "        [   855],\n",
      "        [     2],\n",
      "        [   956],\n",
      "        [    16],\n",
      "        [    10],\n",
      "        [    20],\n",
      "        [     4],\n",
      "        [   203],\n",
      "        [     9],\n",
      "        [  3534],\n",
      "        [   494],\n",
      "        [    57],\n",
      "        [ 11381],\n",
      "        [     9],\n",
      "        [    86],\n",
      "        [  3534],\n",
      "        [   422],\n",
      "        [     8],\n",
      "        [     3],\n",
      "        [   234],\n",
      "        [   106],\n",
      "        [    89],\n",
      "        [   153],\n",
      "        [    10],\n",
      "        [   858],\n",
      "        [    40],\n",
      "        [  2350]],\n",
      "\n",
      "       [[   259],\n",
      "        [   938],\n",
      "        [     8],\n",
      "        [  1990],\n",
      "        [  1290],\n",
      "        [     4],\n",
      "        [    23],\n",
      "        [  1163],\n",
      "        [    17],\n",
      "        [   632],\n",
      "        [   265],\n",
      "        [ 23511],\n",
      "        [     5],\n",
      "        [     2],\n",
      "        [168370],\n",
      "        [   128],\n",
      "        [    28],\n",
      "        [   274],\n",
      "        [     6],\n",
      "        [   110],\n",
      "        [  2732],\n",
      "        [  4024],\n",
      "        [    29],\n",
      "        [   219],\n",
      "        [  3693],\n",
      "        [     2],\n",
      "        [   155],\n",
      "        [    43],\n",
      "        [     3],\n",
      "        [   185]],\n",
      "\n",
      "       [[    99],\n",
      "        [    11],\n",
      "        [    10],\n",
      "        [    20],\n",
      "        [    41],\n",
      "        [     3],\n",
      "        [   169],\n",
      "        [     5],\n",
      "        [   488],\n",
      "        [  1107],\n",
      "        [  3230],\n",
      "        [     9],\n",
      "        [    98],\n",
      "        [    12],\n",
      "        [   128],\n",
      "        [    45],\n",
      "        [    51],\n",
      "        [  1278],\n",
      "        [    68],\n",
      "        [    12],\n",
      "        [    41],\n",
      "        [    71],\n",
      "        [   312],\n",
      "        [    19],\n",
      "        [ 23912],\n",
      "        [    13],\n",
      "        [   792],\n",
      "        [   111],\n",
      "        [  1121],\n",
      "        [ 29825]]]), array([[0],\n",
      "       [1],\n",
      "       [1]]))\n",
      "(array([[[33210],\n",
      "        [21943],\n",
      "        [  202],\n",
      "        [80559],\n",
      "        [ 6175],\n",
      "        [  554],\n",
      "        [    2],\n",
      "        [20365],\n",
      "        [    5],\n",
      "        [   21],\n",
      "        [  121],\n",
      "        [    4],\n",
      "        [  146],\n",
      "        [  143],\n",
      "        [ 4350],\n",
      "        [   47],\n",
      "        [    5],\n",
      "        [  112],\n",
      "        [    8],\n",
      "        [  996],\n",
      "        [    4],\n",
      "        [  470],\n",
      "        [    2],\n",
      "        [  415],\n",
      "        [   27],\n",
      "        [ 2088],\n",
      "        [   30],\n",
      "        [    5],\n",
      "        [    2],\n",
      "        [   78]],\n",
      "\n",
      "       [[    9],\n",
      "        [  273],\n",
      "        [45504],\n",
      "        [90329],\n",
      "        [   30],\n",
      "        [  394],\n",
      "        [    4],\n",
      "        [  199],\n",
      "        [   12],\n",
      "        [   14],\n",
      "        [40236],\n",
      "        [   18],\n",
      "        [ 1109],\n",
      "        [ 2575],\n",
      "        [  288],\n",
      "        [ 6865],\n",
      "        [ 2523],\n",
      "        [ 3379],\n",
      "        [ 1681],\n",
      "        [ 3890],\n",
      "        [   87],\n",
      "        [ 1297],\n",
      "        [   40],\n",
      "        [    2],\n",
      "        [31881],\n",
      "        [  431],\n",
      "        [ 3221],\n",
      "        [    5],\n",
      "        [    3],\n",
      "        [  888]],\n",
      "\n",
      "       [[    8],\n",
      "        [    2],\n",
      "        [ 1430],\n",
      "        [ 4341],\n",
      "        [   10],\n",
      "        [    7],\n",
      "        [    3],\n",
      "        [ 2826],\n",
      "        [  552],\n",
      "        [  117],\n",
      "        [  248],\n",
      "        [   12],\n",
      "        [ 5740],\n",
      "        [    3],\n",
      "        [  723],\n",
      "        [  439],\n",
      "        [  751],\n",
      "        [   19],\n",
      "        [    2],\n",
      "        [ 4812],\n",
      "        [  289],\n",
      "        [10487],\n",
      "        [    9],\n",
      "        [   64],\n",
      "        [   98],\n",
      "        [    5],\n",
      "        [  156],\n",
      "        [  678],\n",
      "        [ 1677],\n",
      "        [   11]]]), array([[1],\n",
      "       [1],\n",
      "       [0]]))\n",
      "(array([[[   10],\n",
      "        [   14],\n",
      "        [ 1996],\n",
      "        [49513],\n",
      "        [76515],\n",
      "        [   24],\n",
      "        [    4],\n",
      "        [   21],\n",
      "        [ 6130],\n",
      "        [    4],\n",
      "        [   27],\n",
      "        [ 1363],\n",
      "        [    5],\n",
      "        [10259],\n",
      "        [ 3611],\n",
      "        [  734],\n",
      "        [   94],\n",
      "        [ 1299],\n",
      "        [   14],\n",
      "        [16102],\n",
      "        [   35],\n",
      "        [    5],\n",
      "        [   63],\n",
      "        [  146],\n",
      "        [   21],\n",
      "        [ 1563],\n",
      "        [  150],\n",
      "        [    8],\n",
      "        [    2],\n",
      "        [   20]],\n",
      "\n",
      "       [[    9],\n",
      "        [  236],\n",
      "        [   10],\n",
      "        [    6],\n",
      "        [   28],\n",
      "        [    2],\n",
      "        [   78],\n",
      "        [  890],\n",
      "        [ 7855],\n",
      "        [  155],\n",
      "        [   77],\n",
      "        [    9],\n",
      "        [  361],\n",
      "        [   12],\n",
      "        [   14],\n",
      "        [    2],\n",
      "        [   78],\n",
      "        [ 6667],\n",
      "        [    4],\n",
      "        [   62],\n",
      "        [    2],\n",
      "        [  116],\n",
      "        [  690],\n",
      "        [ 3221],\n",
      "        [   19],\n",
      "        [   78],\n",
      "        [    5],\n",
      "        [   83],\n",
      "        [ 3398],\n",
      "        [    2]],\n",
      "\n",
      "       [[   10],\n",
      "        [   20],\n",
      "        [  287],\n",
      "        [   79],\n",
      "        [ 4641],\n",
      "        [    2],\n",
      "        [ 6779],\n",
      "        [   61],\n",
      "        [ 2502],\n",
      "        [   89],\n",
      "        [   36],\n",
      "        [   85],\n",
      "        [   23],\n",
      "        [  171],\n",
      "        [    6],\n",
      "        [   28],\n",
      "        [  644],\n",
      "        [   26],\n",
      "        [   60],\n",
      "        [  142],\n",
      "        [    8],\n",
      "        [    2],\n",
      "        [  108],\n",
      "        [   59],\n",
      "        [  517],\n",
      "        [   22],\n",
      "        [79788],\n",
      "        [   10],\n",
      "        [   20],\n",
      "        [  287]]]), array([[1],\n",
      "       [1],\n",
      "       [1]]))\n",
      "(array([[[    10],\n",
      "        [     7],\n",
      "        [     3],\n",
      "        [   982],\n",
      "        [  1739],\n",
      "        [   350],\n",
      "        [  1185],\n",
      "        [     5],\n",
      "        [222123],\n",
      "        [   409],\n",
      "        [222124],\n",
      "        [     2],\n",
      "        [    75],\n",
      "        [     7],\n",
      "        [   427],\n",
      "        [    19],\n",
      "        [222125],\n",
      "        [222126],\n",
      "        [   368],\n",
      "        [     5],\n",
      "        [     2],\n",
      "        [   163],\n",
      "        [   451],\n",
      "        [     4],\n",
      "        [    12],\n",
      "        [     7],\n",
      "        [     3],\n",
      "        [ 26644],\n",
      "        [     5],\n",
      "        [     2]],\n",
      "\n",
      "       [[     9],\n",
      "        [    26],\n",
      "        [   119],\n",
      "        [    10],\n",
      "        [    24],\n",
      "        [  1837],\n",
      "        [   286],\n",
      "        [     4],\n",
      "        [    16],\n",
      "        [     2],\n",
      "        [   161],\n",
      "        [     5],\n",
      "        [   493],\n",
      "        [     9],\n",
      "        [   486],\n",
      "        [   362],\n",
      "        [   144],\n",
      "        [    45],\n",
      "        [    89],\n",
      "        [  1633],\n",
      "        [    10],\n",
      "        [     6],\n",
      "        [ 31892],\n",
      "        [    10],\n",
      "        [    24],\n",
      "        [     7],\n",
      "        [    23],\n",
      "        [    43],\n",
      "        [     2],\n",
      "        [  1093]],\n",
      "\n",
      "       [[     9],\n",
      "        [    61],\n",
      "        [   526],\n",
      "        [   995],\n",
      "        [     6],\n",
      "        [    67],\n",
      "        [  1713],\n",
      "        [     5],\n",
      "        [     2],\n",
      "        [ 15969],\n",
      "        [    18],\n",
      "        [    12],\n",
      "        [    14],\n",
      "        [     3],\n",
      "        [   588],\n",
      "        [ 92440],\n",
      "        [    13],\n",
      "        [    93],\n",
      "        [  3933],\n",
      "        [     4],\n",
      "        [  7673],\n",
      "        [    22],\n",
      "        [  1160],\n",
      "        [    18],\n",
      "        [    11],\n",
      "        [     7],\n",
      "        [     2],\n",
      "        [    58],\n",
      "        [    56],\n",
      "        [  1310]]]), array([[0],\n",
      "       [1],\n",
      "       [0]]))\n",
      "(array([[[194988],\n",
      "        [    48],\n",
      "        [     3],\n",
      "        [  8789],\n",
      "        [  5509],\n",
      "        [ 41423],\n",
      "        [    14],\n",
      "        [     3],\n",
      "        [  1406],\n",
      "        [    17],\n",
      "        [    10],\n",
      "        [   139],\n",
      "        [    27],\n",
      "        [    62],\n",
      "        [     2],\n",
      "        [  4850],\n",
      "        [     6],\n",
      "        [  2294],\n",
      "        [     3],\n",
      "        [    24],\n",
      "        [    17],\n",
      "        [   184],\n",
      "        [    11],\n",
      "        [   367],\n",
      "        [  3281],\n",
      "        [    19],\n",
      "        [   426],\n",
      "        [     5],\n",
      "        [   419],\n",
      "        [    27]],\n",
      "\n",
      "       [[   640],\n",
      "        [ 13802],\n",
      "        [  1074],\n",
      "        [     2],\n",
      "        [  1088],\n",
      "        [     4],\n",
      "        [    95],\n",
      "        [    21],\n",
      "        [  3623],\n",
      "        [  2557],\n",
      "        [    17],\n",
      "        [ 65495],\n",
      "        [     5],\n",
      "        [ 88206],\n",
      "        [     3],\n",
      "        [   122],\n",
      "        [  2278],\n",
      "        [ 21339],\n",
      "        [    17],\n",
      "        [  1962],\n",
      "        [ 26281],\n",
      "        [     8],\n",
      "        [    63],\n",
      "        [     3],\n",
      "        [  4669],\n",
      "        [     7],\n",
      "        [ 12241],\n",
      "        [    72],\n",
      "        [     2],\n",
      "        [   513]],\n",
      "\n",
      "       [[   471],\n",
      "        [   446],\n",
      "        [    23],\n",
      "        [    99],\n",
      "        [     2],\n",
      "        [   225],\n",
      "        [    70],\n",
      "        [    12],\n",
      "        [  1773],\n",
      "        [   161],\n",
      "        [    15],\n",
      "        [     3],\n",
      "        [   304],\n",
      "        [     8],\n",
      "        [  1304],\n",
      "        [  3661],\n",
      "        [    70],\n",
      "        [    41],\n",
      "        [ 48387],\n",
      "        [     2],\n",
      "        [   388],\n",
      "        [ 24109],\n",
      "        [    11],\n",
      "        [  1466],\n",
      "        [113342],\n",
      "        [   209],\n",
      "        [  1185],\n",
      "        [    17],\n",
      "        [  1086],\n",
      "        [   787]]]), array([[0],\n",
      "       [1],\n",
      "       [1]]))\n"
     ]
    }
   ],
   "source": [
    "#编写一个迭代器，每次调用这个迭代器都会返回一个新的batch，用于训练或者预测\n",
    "def build_batch(word2id_dict, corpus, batch_size, epoch_num, max_seq_len, shuffle = True):\n",
    "\n",
    "    #模型将会接受的两个输入：\n",
    "    # 1. 一个形状为[batch_size, max_seq_len]的张量，sentence_batch，代表了一个mini-batch的句子。\n",
    "    # 2. 一个形状为[batch_size, 1]的张量，sentence_label_batch，\n",
    "    #    每个元素都是非0即1，代表了每个句子的情感类别（正向或者负向）\n",
    "    sentence_batch = []\n",
    "    sentence_label_batch = []\n",
    "\n",
    "    for _ in range(epoch_num): \n",
    "\n",
    "        #每个epcoh前都shuffle一下数据，有助于提高模型训练的效果\n",
    "        #但是对于预测任务，不要做数据shuffle\n",
    "        if shuffle:\n",
    "            random.shuffle(corpus)\n",
    "\n",
    "        for sentence, sentence_label in corpus:\n",
    "            sentence_sample = sentence[:min(max_seq_len, len(sentence))]\n",
    "            if len(sentence_sample) < max_seq_len:\n",
    "                for _ in range(max_seq_len - len(sentence_sample)):\n",
    "                    sentence_sample.append(word2id_dict['[pad]'])\n",
    "            \n",
    "            #飞桨在1.6.1要求输入数据必须是形状为[batch_size, max_seq_len，1]的张量\n",
    "            #在飞桨的后续版本中不再存在类似的要求\n",
    "            sentence_sample = [[word_id] for word_id in sentence_sample]\n",
    "\n",
    "            sentence_batch.append(sentence_sample)\n",
    "            sentence_label_batch.append([sentence_label])\n",
    "\n",
    "            if len(sentence_batch) == batch_size:\n",
    "                yield np.array(sentence_batch).astype(\"int64\"), np.array(sentence_label_batch).astype(\"int64\")\n",
    "                sentence_batch = []\n",
    "                sentence_label_batch = []\n",
    "\n",
    "    if len(sentence_batch) == batch_size:\n",
    "        yield np.array(sentence_batch).astype(\"int64\"), np.array(sentence_label_batch).astype(\"int64\")\n",
    "\n",
    "for _, batch in zip(range(10), build_batch(word2id_dict, \n",
    "                    train_corpus, batch_size=3, epoch_num=3, max_seq_len=30)):\n",
    "    print(batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 网络定义\n",
    "\n",
    "#### 1. 定义长短时记忆模型\n",
    "\n",
    "使用飞桨定义一个长短时记忆模型，以便情感分析模型调用，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#使用飞桨实现一个长短时记忆模型\n",
    "class SimpleLSTMRNN(fluid.Layer):\n",
    "    \n",
    "    def __init__(self,\n",
    "                 name_scope,\n",
    "                 hidden_size,\n",
    "                 num_steps,\n",
    "                 num_layers=1,\n",
    "                 init_scale=0.1,\n",
    "                 dropout=None):\n",
    "        \n",
    "        #这个模型有几个参数：\n",
    "        #1. name_scope，表示这个模型的名字，用于区别不同的实例\n",
    "        #2. hidden_size，表示embedding-size，或者是记忆向量的维度\n",
    "        #3. num_steps，表示这个长短时记忆网络，最多可以考虑多长的时间序列\n",
    "        #4. num_layers，表示这个长短时记忆网络内部有多少层，我们知道，\n",
    "        #   给定一个形状为[batch_size, seq_len, embedding_size]的输入，\n",
    "        # 长短时记忆网络会输出一个同样为[batch_size, seq_len, embedding_size]的输出，\n",
    "        #   我们可以把这个输出再链到一个新的长短时记忆网络上\n",
    "        # 如此叠加多层长短时记忆网络，有助于学习更复杂的句子甚至是篇章。\n",
    "        #5. init_scale，表示网络内部的参数的初始化范围，\n",
    "        # 长短时记忆网络内部用了很多tanh，sigmoid等激活函数，这些函数对数值精度非常敏感，\n",
    "        # 因此我们一般只使用比较小的初始化范围，以保证效果，\n",
    "        \n",
    "        super(SimpleLSTMRNN, self).__init__(name_scope)\n",
    "        self._hidden_size = hidden_size\n",
    "        self._num_layers = num_layers\n",
    "        self._init_scale = init_scale\n",
    "        self._dropout = dropout\n",
    "        self._input = None\n",
    "        self._num_steps = num_steps\n",
    "        self.cell_array = []\n",
    "        self.hidden_array = []\n",
    "\n",
    "        # weight_1_arr用于存储不同层的长短时记忆网络中，不同门的W参数\n",
    "        self.weight_1_arr = []\n",
    "        self.weight_2_arr = []\n",
    "        # bias_arr用于存储不同层的长短时记忆网络中，不同门的b参数\n",
    "        self.bias_arr = []\n",
    "        self.mask_array = []\n",
    "\n",
    "        # 通过使用create_parameter函数，创建不同长短时记忆网络层中的参数\n",
    "        # 通过上面的公式，我们知道，我们总共需要8个形状为[_hidden_size, _hidden_size]的W向量\n",
    "        # 和4个形状为[_hidden_size]的b向量，因此，我们在声明参数的时候，\n",
    "        # 一次性声明一个大小为[self._hidden_size * 2, self._hidden_size * 4]的参数\n",
    "        # 和一个 大小为[self._hidden_size * 4]的参数，这样做的好处是，\n",
    "        # 可以使用一次矩阵计算，同时计算8个不同的矩阵乘法\n",
    "        # 以便加快计算速度\n",
    "        for i in range(self._num_layers):\n",
    "            weight_1 = self.create_parameter(\n",
    "                attr=fluid.ParamAttr(\n",
    "                    initializer=fluid.initializer.UniformInitializer(\n",
    "                        low=-self._init_scale, high=self._init_scale)),\n",
    "                shape=[self._hidden_size * 2, self._hidden_size * 4],\n",
    "                dtype=\"float32\",\n",
    "                default_initializer=fluid.initializer.UniformInitializer(\n",
    "                    low=-self._init_scale, high=self._init_scale))\n",
    "            self.weight_1_arr.append(self.add_parameter('w_%d' % i, weight_1))\n",
    "            bias_1 = self.create_parameter(\n",
    "                attr=fluid.ParamAttr(\n",
    "                    initializer=fluid.initializer.UniformInitializer(\n",
    "                        low=-self._init_scale, high=self._init_scale)),\n",
    "                shape=[self._hidden_size * 4],\n",
    "                dtype=\"float32\",\n",
    "                default_initializer=fluid.initializer.Constant(0.0))\n",
    "            self.bias_arr.append(self.add_parameter('b_%d' % i, bias_1))\n",
    "\n",
    "    # 定义LSTM网络的前向计算逻辑，飞桨会自动根据前向计算结果，给出反向结果\n",
    "    def forward(self, input_embedding, init_hidden=None, init_cell=None):\n",
    "        self.cell_array = []\n",
    "        self.hidden_array = []\n",
    "        \n",
    "        #输入有三个信号：\n",
    "        # 1. input_embedding，这个就是输入句子的embedding表示，\n",
    "        # 是一个形状为[batch_size, seq_len, embedding_size]的张量\n",
    "        # 2. init_hidden，这个表示LSTM中每一层的初始h的值，有时候，\n",
    "        # 我们需要显示地指定这个值，在不需要的时候，就可以把这个值设置为空\n",
    "        # 3. init_cell，这个表示LSTM中每一层的初始c的值，有时候，\n",
    "        # 我们需要显示地指定这个值，在不需要的时候，就可以把这个值设置为空\n",
    "\n",
    "        # 我们需要通过slice操作，把每一层的初始hidden和cell值拿出来，\n",
    "        # 并存储在cell_array和hidden_array中\n",
    "        for i in range(self._num_layers):\n",
    "            pre_hidden = fluid.layers.slice(\n",
    "                init_hidden, axes=[0], starts=[i], ends=[i + 1])\n",
    "            pre_cell = fluid.layers.slice(\n",
    "                init_cell, axes=[0], starts=[i], ends=[i + 1])\n",
    "            pre_hidden = fluid.layers.reshape(\n",
    "                pre_hidden, shape=[-1, self._hidden_size])\n",
    "            pre_cell = fluid.layers.reshape(\n",
    "                pre_cell, shape=[-1, self._hidden_size])\n",
    "            self.hidden_array.append(pre_hidden)\n",
    "            self.cell_array.append(pre_cell)\n",
    "\n",
    "        # res记录了LSTM中每一层的输出结果（hidden）\n",
    "        res = []\n",
    "        for index in range(self._num_steps):\n",
    "            # 首先需要通过slice函数，拿到输入tensor input_embedding中当前位置的词的向量表示\n",
    "            # 并把这个词的向量表示转换为一个大小为 [batch_size, embedding_size]的张量\n",
    "            self._input = fluid.layers.slice(\n",
    "                input_embedding, axes=[1], starts=[index], ends=[index + 1])\n",
    "            self._input = fluid.layers.reshape(\n",
    "                self._input, shape=[-1, self._hidden_size])\n",
    "            \n",
    "            # 计算每一层的结果，从下而上\n",
    "            for k in range(self._num_layers):\n",
    "                # 首先获取每一层LSTM对应上一个时间步的hidden，cell，以及当前层的W和b参数\n",
    "                pre_hidden = self.hidden_array[k]\n",
    "                pre_cell = self.cell_array[k]\n",
    "                weight_1 = self.weight_1_arr[k]\n",
    "                bias = self.bias_arr[k]\n",
    "\n",
    "                # 我们把hidden和拿到的当前步的input拼接在一起，便于后续计算\n",
    "                nn = fluid.layers.concat([self._input, pre_hidden], 1)\n",
    "                \n",
    "                # 将输入门，遗忘门，输出门等对应的W参数，和输入input和pre-hidden相乘\n",
    "                # 我们通过一步计算，就同时完成了8个不同的矩阵运算，提高了运算效率\n",
    "                gate_input = fluid.layers.matmul(x=nn, y=weight_1)\n",
    "\n",
    "                # 将b参数也加入到前面的运算结果中\n",
    "                gate_input = fluid.layers.elementwise_add(gate_input, bias)\n",
    "                \n",
    "                # 通过split函数，将每个门得到的结果拿出来\n",
    "                i, j, f, o = fluid.layers.split(\n",
    "                    gate_input, num_or_sections=4, dim=-1)\n",
    "                \n",
    "                # 把输入门，遗忘门，输出门等对应的权重作用在当前输入input和pre-hidden上\n",
    "                c = pre_cell * fluid.layers.sigmoid(f) + fluid.layers.sigmoid(\n",
    "                    i) * fluid.layers.tanh(j)\n",
    "                m = fluid.layers.tanh(c) * fluid.layers.sigmoid(o)\n",
    "                \n",
    "                # 记录当前步骤的计算结果，\n",
    "                # m是当前步骤需要输出的hidden\n",
    "                # c是当前步骤需要输出的cell\n",
    "                self.hidden_array[k] = m\n",
    "                self.cell_array[k] = c\n",
    "                self._input = m\n",
    "                \n",
    "                # 一般来说，我们有时候会在LSTM的结果结果内加入dropout操作\n",
    "                # 这样会提高模型的训练鲁棒性\n",
    "                if self._dropout is not None and self._dropout > 0.0:\n",
    "                    self._input = fluid.layers.dropout(\n",
    "                        self._input,\n",
    "                        dropout_prob=self._dropout,\n",
    "                        dropout_implementation='upscale_in_train')\n",
    "                    \n",
    "            res.append(\n",
    "                fluid.layers.reshape(\n",
    "                    self._input, shape=[1, -1, self._hidden_size]))\n",
    "        \n",
    "        # 计算长短时记忆网络的结果返回回来，包括：\n",
    "        # 1. real_res：每个时间步上不同层的hidden结果\n",
    "        # 2. last_hidden：最后一个时间步中，每一层的hidden的结果，\n",
    "        # 形状为：[batch_size, num_layers, hidden_size]\n",
    "        # 3. last_cell：最后一个时间步中，每一层的cell的结果，\n",
    "        # 形状为：[batch_size, num_layers, hidden_size]\n",
    "        real_res = fluid.layers.concat(res, 0)\n",
    "        real_res = fluid.layers.transpose(x=real_res, perm=[1, 0, 2])\n",
    "        last_hidden = fluid.layers.concat(self.hidden_array, 1)\n",
    "        last_hidden = fluid.layers.reshape(\n",
    "            last_hidden, shape=[-1, self._num_layers, self._hidden_size])\n",
    "        last_hidden = fluid.layers.transpose(x=last_hidden, perm=[1, 0, 2])\n",
    "        last_cell = fluid.layers.concat(self.cell_array, 1)\n",
    "        last_cell = fluid.layers.reshape(\n",
    "            last_cell, shape=[-1, self._num_layers, self._hidden_size])\n",
    "        last_cell = fluid.layers.transpose(x=last_cell, perm=[1, 0, 2])\n",
    "        \n",
    "        return real_res, last_hidden, last_cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. 定义情感分析模型\n",
    "\n",
    "长短时记忆模型定义完成后，我们便可以开始定义一个基于长短时记忆模型的情感分析模型，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义一个可以用于情感分类的网络\n",
    "class SentimentClassifier(fluid.Layer):\n",
    "    def __init__(self,\n",
    "                 name_scope,\n",
    "                 hidden_size,\n",
    "                 vocab_size,\n",
    "                 class_num=2,\n",
    "                 num_layers=1,\n",
    "                 num_steps=128,\n",
    "                 init_scale=0.1,\n",
    "                 dropout=None):\n",
    "        \n",
    "        #这个模型的参数分别为：\n",
    "        #1. name_scope，表示这个模型的名字，用于区别不同的实例\n",
    "        #2. hidden_size，表示embedding-size，hidden已经cell向量的维度\n",
    "        #3. vocab_size，模型可以考虑的词表大小\n",
    "        #4. class_num，情感类型个数，可以是2分类，也可以是多分类\n",
    "        #5. num_steps，表示这个情感分析模型最大可以考虑的句子长度\n",
    "        #6. init_scale，表示网络内部的参数的初始化范围，\n",
    "        # 长短时记忆网络内部用了很多tanh，sigmoid等激活函数，这些函数对数值精度非常敏感，\n",
    "        # 因此我们一般只使用比较小的初始化范围，以保证效果\n",
    "        \n",
    "        super(SentimentClassifier, self).__init__(name_scope)\n",
    "        self.hidden_size = hidden_size\n",
    "        self.vocab_size = vocab_size\n",
    "        self.class_num = class_num\n",
    "        self.init_scale = init_scale\n",
    "        self.num_layers = num_layers\n",
    "        self.num_steps = num_steps\n",
    "        self.dropout = dropout\n",
    "\n",
    "        # 声明一个LSTM模型，用来把一个句子抽象城一个向量\n",
    "        self.simple_lstm_rnn = SimpleLSTMRNN(\n",
    "            self.full_name(),\n",
    "            hidden_size,\n",
    "            num_steps,\n",
    "            num_layers=num_layers,\n",
    "            init_scale=init_scale,\n",
    "            dropout=dropout)\n",
    "        \n",
    "        # 声明一个embedding层，用来把句子中的每个词转换为向量\n",
    "        self.embedding = Embedding(\n",
    "            self.full_name(),\n",
    "            size=[vocab_size, hidden_size],\n",
    "            dtype='float32',\n",
    "            is_sparse=False,\n",
    "            param_attr=fluid.ParamAttr(\n",
    "                name='embedding_para',\n",
    "                initializer=fluid.initializer.UniformInitializer(\n",
    "                    low=-init_scale, high=init_scale)))\n",
    "        \n",
    "        # 在得到一个句子的向量表示后，我们需要根据这个向量表示对这个句子进行分类\n",
    "        # 一般来说，我们可以把这个句子的向量表示，\n",
    "        # 乘以一个大小为[self.hidden_size, self.class_num]的W参数\n",
    "        # 并加上一个大小为[self.class_num]的b参数\n",
    "        # 通过这种手段达到把句子向量映射到分类结果的目标\n",
    "        \n",
    "        # 我们需要声明最终在使用句子向量映射到具体情感类别过程中所需要使用的参数\n",
    "        # 这个参数的大小一般是[self.hidden_size, self.class_num]\n",
    "        self.softmax_weight = self.create_parameter(\n",
    "            attr=fluid.ParamAttr(),\n",
    "            shape=[self.hidden_size, self.class_num],\n",
    "            dtype=\"float32\",\n",
    "            default_initializer=fluid.initializer.UniformInitializer(\n",
    "                low=-self.init_scale, high=self.init_scale))\n",
    "        # 同样的，我们需要声明最终分类过程中的b参数\n",
    "        #  这个参数的大小一般是[self.class_num]\n",
    "        self.softmax_bias = self.create_parameter(\n",
    "            attr=fluid.ParamAttr(),\n",
    "            shape=[self.class_num],\n",
    "            dtype=\"float32\",\n",
    "            default_initializer=fluid.initializer.UniformInitializer(\n",
    "                low=-self.init_scale, high=self.init_scale))\n",
    "\n",
    "    def forward(self, input, label):\n",
    "\n",
    "        # 首先我们需要定义LSTM的初始hidden和cell，这里我们使用0来初始化这个序列的记忆\n",
    "        init_hidden_data = np.zeros(\n",
    "            (1, batch_size, embedding_size), dtype='float32')\n",
    "        init_cell_data = np.zeros(\n",
    "            (1, batch_size, embedding_size), dtype='float32')\n",
    "\n",
    "        # 将这些初始记忆转换为飞桨可计算的向量\n",
    "        # 并设置stop-gradient=True，避免这些向量被更新，从而影响训练效果\n",
    "        init_hidden = fluid.dygraph.to_variable(init_hidden_data)\n",
    "        init_hidden.stop_gradient = True\n",
    "        init_cell = fluid.dygraph.to_variable(init_cell_data)\n",
    "        init_cell.stop_gradient = True\n",
    "\n",
    "        init_h = fluid.layers.reshape(\n",
    "            init_hidden, shape=[self.num_layers, -1, self.hidden_size])\n",
    "\n",
    "        init_c = fluid.layers.reshape(\n",
    "            init_cell, shape=[self.num_layers, -1, self.hidden_size])\n",
    "\n",
    "        # 将输入的句子的mini-batch input，转换为词向量表示\n",
    "        x_emb = self.embedding(input)\n",
    "\n",
    "        x_emb = fluid.layers.reshape(\n",
    "            x_emb, shape=[-1, self.num_steps, self.hidden_size])\n",
    "        if self.dropout is not None and self.dropout > 0.0:\n",
    "            x_emb = fluid.layers.dropout(\n",
    "                x_emb,\n",
    "                dropout_prob=self.dropout,\n",
    "                dropout_implementation='upscale_in_train')\n",
    "        \n",
    "        # 使用LSTM网络，把每个句子转换为向量表示\n",
    "        rnn_out, last_hidden, last_cell = self.simple_lstm_rnn(x_emb, init_h,\n",
    "                                                               init_c)\n",
    "        last_hidden = fluid.layers.reshape(\n",
    "            last_hidden, shape=[-1, self.hidden_size])\n",
    "        \n",
    "        # 将每个句子的向量表示，通过矩阵计算，映射到具体的情感类别上\n",
    "        projection = fluid.layers.matmul(last_hidden, self.softmax_weight)\n",
    "        projection = fluid.layers.elementwise_add(projection, self.softmax_bias)\n",
    "        projection = fluid.layers.reshape(\n",
    "            projection, shape=[-1, self.class_num])\n",
    "        pred = fluid.layers.softmax(projection, axis=-1)\n",
    "        \n",
    "        # 根据给定的标签信息，计算整个网络的损失函数，这里我们可以直接使用分类任务中常使用的交叉熵来训练网络\n",
    "        loss = fluid.layers.softmax_with_cross_entropy(\n",
    "            logits=projection, label=label, soft_label=False)\n",
    "        loss = fluid.layers.reduce_mean(loss)\n",
    "\n",
    "        # 最终返回预测结果pred，和网络的loss\n",
    "        return pred, loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型训练\n",
    "\n",
    "在完成模型定义之后，我们就可以开始训练模型了。当训练结束以后，我们可以使用测试集合评估一下当前模型的效果，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 10, loss 0.701\n",
      "step 20, loss 0.718\n",
      "step 30, loss 0.761\n",
      "step 40, loss 0.687\n",
      "step 50, loss 0.681\n",
      "step 60, loss 0.696\n",
      "step 70, loss 0.689\n",
      "step 80, loss 0.706\n",
      "step 90, loss 0.682\n",
      "step 100, loss 0.704\n",
      "step 110, loss 0.680\n",
      "step 120, loss 0.645\n",
      "step 130, loss 0.533\n",
      "step 140, loss 0.628\n",
      "step 150, loss 0.624\n",
      "step 160, loss 0.591\n",
      "step 170, loss 0.515\n",
      "step 180, loss 0.422\n",
      "step 190, loss 0.417\n",
      "step 200, loss 0.332\n",
      "step 210, loss 0.432\n",
      "step 220, loss 0.403\n",
      "step 230, loss 0.307\n",
      "step 240, loss 0.313\n",
      "step 250, loss 0.399\n",
      "step 260, loss 0.371\n",
      "step 270, loss 0.275\n",
      "step 280, loss 0.328\n",
      "step 290, loss 0.339\n",
      "step 300, loss 0.317\n",
      "step 310, loss 0.329\n",
      "step 320, loss 0.272\n",
      "step 330, loss 0.259\n",
      "step 340, loss 0.257\n",
      "step 350, loss 0.297\n",
      "step 360, loss 0.242\n",
      "step 370, loss 0.393\n",
      "step 380, loss 0.360\n",
      "step 390, loss 0.361\n",
      "step 400, loss 0.053\n",
      "step 410, loss 0.024\n",
      "step 420, loss 0.077\n",
      "step 430, loss 0.095\n",
      "step 440, loss 0.029\n",
      "step 450, loss 0.049\n",
      "step 460, loss 0.053\n",
      "step 470, loss 0.121\n",
      "step 480, loss 0.048\n",
      "step 490, loss 0.023\n",
      "step 500, loss 0.115\n",
      "step 510, loss 0.091\n",
      "step 520, loss 0.038\n",
      "step 530, loss 0.036\n",
      "step 540, loss 0.044\n",
      "step 550, loss 0.065\n",
      "step 560, loss 0.081\n",
      "step 570, loss 0.093\n",
      "step 580, loss 0.046\n",
      "step 590, loss 0.031\n",
      "step 600, loss 0.039\n",
      "step 610, loss 0.007\n",
      "step 620, loss 0.002\n",
      "step 630, loss 0.007\n",
      "step 640, loss 0.063\n",
      "step 650, loss 0.018\n",
      "step 660, loss 0.015\n",
      "step 670, loss 0.007\n",
      "step 680, loss 0.006\n",
      "step 690, loss 0.006\n",
      "step 700, loss 0.013\n",
      "step 710, loss 0.053\n",
      "step 720, loss 0.107\n",
      "step 730, loss 0.032\n",
      "step 740, loss 0.007\n",
      "step 750, loss 0.007\n",
      "step 760, loss 0.017\n",
      "step 770, loss 0.017\n",
      "step 780, loss 0.010\n",
      "step 790, loss 0.002\n",
      "step 800, loss 0.018\n",
      "step 810, loss 0.006\n",
      "step 820, loss 0.046\n",
      "step 830, loss 0.001\n",
      "step 840, loss 0.002\n",
      "step 850, loss 0.002\n",
      "step 860, loss 0.001\n",
      "step 870, loss 0.001\n",
      "step 880, loss 0.008\n",
      "step 890, loss 0.001\n",
      "step 900, loss 0.004\n",
      "step 910, loss 0.021\n",
      "step 920, loss 0.000\n",
      "step 930, loss 0.003\n",
      "step 940, loss 0.001\n",
      "step 950, loss 0.002\n",
      "step 960, loss 0.001\n",
      "step 970, loss 0.006\n",
      "the acc in the test set is 0.764\n"
     ]
    }
   ],
   "source": [
    "import paddle.fluid as fluid\n",
    "#开始训练\n",
    "batch_size = 128\n",
    "epoch_num = 5\n",
    "embedding_size = 256\n",
    "step = 0\n",
    "learning_rate = 0.01\n",
    "max_seq_len = 128\n",
    "\n",
    "with fluid.dygraph.guard(fluid.CUDAPlace(0)):\n",
    "    # 创建一个用于情感分类的网络实例，sentiment_classifier\n",
    "    sentiment_classifier = SentimentClassifier(\n",
    "        \"sentiment_classifier\", embedding_size, vocab_size, num_steps=max_seq_len)\n",
    "    # 创建优化器AdamOptimizer，用于更新这个网络的参数\n",
    "    adam = fluid.optimizer.AdamOptimizer(learning_rate=learning_rate)\n",
    "\n",
    "    for sentences, labels in build_batch(\n",
    "        word2id_dict, train_corpus, batch_size, epoch_num, max_seq_len):\n",
    "        \n",
    "        sentences_var = fluid.dygraph.to_variable(sentences)\n",
    "        labels_var = fluid.dygraph.to_variable(labels)\n",
    "        pred, loss = sentiment_classifier(sentences_var, labels_var)\n",
    "\n",
    "        loss.backward()\n",
    "        adam.minimize(loss)\n",
    "        sentiment_classifier.clear_gradients()\n",
    "        \n",
    "        step += 1\n",
    "        if step % 10 == 0:\n",
    "            print(\"step %d, loss %.3f\" % (step, loss.numpy()[0]))\n",
    "            \n",
    "    # 我们希望在网络训练结束以后评估一下训练好的网络的效果\n",
    "    # 通过eval()函数，将网络设置为eval模式，在eval模式中，网络不会进行梯度更新\n",
    "    sentiment_classifier.eval()\n",
    "    # 这里我们需要记录模型预测结果的准确率\n",
    "    # 对于二分类任务来说，准确率的计算公式为：\n",
    "    # (true_positive + true_negative) / \n",
    "    # (true_positive + true_negative + false_positive + false_negative)\n",
    "    tp = 0.\n",
    "    tn = 0.\n",
    "    fp = 0.\n",
    "    fn = 0.\n",
    "    for sentences, labels in build_batch(\n",
    "        word2id_dict, test_corpus, batch_size, 1, max_seq_len):\n",
    "        \n",
    "        sentences_var = fluid.dygraph.to_variable(sentences)\n",
    "        labels_var = fluid.dygraph.to_variable(labels)\n",
    "        \n",
    "        # 获取模型对当前batch的输出结果\n",
    "        pred, loss = sentiment_classifier(sentences_var, labels_var)\n",
    "\n",
    "        # 把输出结果转换为numpy array的数据结构\n",
    "        # 遍历这个数据结构，比较预测结果和对应label之间的关系，并更新tp，tn，fp和fn\n",
    "        pred = pred.numpy()\n",
    "        for i in range(len(pred)):\n",
    "            if labels[i][0] == 1:\n",
    "                if pred[i][1] > pred[i][0]:\n",
    "                    tp += 1\n",
    "                else:\n",
    "                    fn += 1\n",
    "            else:\n",
    "                if pred[i][1] > pred[i][0]:\n",
    "                    fp += 1\n",
    "                else:\n",
    "                    tn += 1\n",
    "\n",
    "    # 输出最终评估的模型效果\n",
    "    print(\"the acc in the test set is %.3f\" % ((tp + tn) / (tp + tn + fp + fn)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本匹配\n",
    "\n",
    "借助相同的思路，我们可以很轻易的解决文本相似度计算问题，假设给定两个句子：\n",
    "\n",
    "> 句子1：我不爱吃烤冷面，但是我爱吃冷面\n",
    ">\n",
    "> 句子2：我爱吃菠萝，但是不爱吃地瓜\n",
    "\n",
    "同样使用LSTM网络，把每个句子抽象成一个向量表示，通过计算这两个向量之间的相似度，就可以快速完成文本相似度计算任务。在实际场景里，我们也通常使用LSTM网络的最后一步hidden结果，将一个句子抽象成一个向量，然后通过向量点积，或者cosine相似度的方式，去衡量两个句子的相似度。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/05f2c05ec0194cd9a53eeec02a92a8304bd679bd153c40cfa0a58cf2f7d905dd\" width=\"400\" ></center>\n",
    "<br><center>图9：文本相似度计算</center></br>\n",
    "\n",
    "一般情况下，在训练阶段有point-wise和pair-wise两个常见的训练模式（针对搜索引擎任务，还有一类list-wise的方法，这里不做探讨）。\n",
    "* **point-wise训练模式：** 在point-wise训练过程中，我们把不同的句子对儿分为两类（或者更多类别）：相似、不相似。通过这种方式把句子相似度计算任务转化为了一个分类问题，通过常见的二分类函数（如sigmoid）即可完成分类任务。在最终预测阶段，使用sigmoid函数的输出，作为两个不同句子的相似度值。\n",
    "\n",
    "* **pair-wise训练模式：** pair-wise训练模式相对更复杂一些，假定给定3个句子，A，B和C。已知A和B相似，但是A和C不相似，那么原则上，A和B的相似度值应该高于A和C的相似度值。因此我们可以构造一个新的训练算法：对于一个相同的相似度计算模型m，假定m(A,B)是m输出的A和B的相似度值，m(A,C)是m输出的A和C的相似度值，那么hinge-loss：\n",
    "$L = \\lambda - (m(A,B)-m(A,C))$ if $m(A,B)-m(A,C) < \\lambda$ else $0$\n",
    "\n",
    "这个损失函数要求对于每个正样本m(A,B)的相似度值至少高于负样本m(A,C)一个阈值$\\lambda$。\n",
    "\n",
    "hinge-loss的好处是没有强迫进行单个样本的分类，而是通过考虑样本和样本直接的大小关系来学习相似和不相似关系。相比较而言，pair-wise训练比point-wise任务效果更加鲁棒一些，更适合如搜索，排序，推荐等场景的相似度计算任务。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 思考一下\n",
    "\n",
    "[1] 情感分析任务对你有什么启发？\n",
    "\n",
    "[2] 除了LSTM，你还能想到那些其他方法，构造一个句子的向量表示？\n",
    "\n",
    "[3] 对一个句子生成一个单一的向量表示有什么缺点，你还知道其他方式吗？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 引用\n",
    "\n",
    "[1] [维基百科：情感分析](https://zh.wikipedia.org/zh-hans/文本情感分析)\n",
    "\n",
    "[2] [维基百科：RNN](https://en.wikipedia.org/wiki/Recurrent_neural_network)\n",
    "\n",
    "[3] [维基百科：LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory)\n",
    "\n",
    "[4] [bilibili：肥伦秀](https://www.bilibili.com/video/av40396494?from=search&seid=9852893210841347755)\n",
    "\n",
    "[5] [知乎：GIF动图一步一步看懂LSTM和GRU](https://zhuanlan.zhihu.com/p/81549798)\n",
    "\n",
    "[6] [Understanding LSTMs](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
